Error in databricks : from ray.train.horovod import HorovodTrainer

1

Using databricks notebook

Screenshot 2023-05-30 at 3.35.21 PM
Screenshot 2023-05-30 at 3.35.21 PM1534×670 51.9 KB

Libraries installed in pip on cluster=
HOROVOD_WITH_GLOO=1 … pip install ‘horovod[ray]’
pip install horovod[ray]
pip install ray[default]>=2.3.0
pip install raydp

Getting below error :-

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
2

Yes,
this is because incompatible protobuf version, just do any of:

  1. Downgrade the protobuf package to 3.20.x or lower.
  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python in Databricks cluster environment setting.