I am trying to run Ray from a docker container on Mac M1 (ubuntu:20.04, built via the command docker buildx build --platform linux/x86_64 -t dev_container -f Dockerfile.dev .)
ray.init() is working fine. I have ray version 2.6.3 and python version 3.9.10. However while executing a remote function with 2 cpus, I am getting the following error
LocalRayletDiedError: The task's local raylet died. Check raylet.out for more information.
output from raylet.out
[2023-09-04 15:21:57,501 I 167 167] (raylet) accessor.cc:611: Received notification for node id = dd8b7120d900b938edc6968bd010b18dde103e7f27d1ebd305320fbd, IsAlive = 1
[2023-09-04 15:21:57,654 I 167 167] (raylet) worker_pool.cc:489: Started worker process with pid 219, the token is 0
[2023-09-04 15:21:57,659 I 167 167] (raylet) worker_pool.cc:489: Started worker process with pid 222, the token is 1
[2023-09-04 15:22:01,932 I 167 186] (raylet) object_store.cc:35: Object store current usage 8e-09 / 0.374179 GB.
[2023-09-04 15:22:01,988 I 167 167] (raylet) node_manager.cc:624: New job has started. Job id 01000000 Driver pid 72 is dead: 0 driver address: ***.**.*.*
[2023-09-04 15:22:01,988 I 167 167] (raylet) worker_pool.cc:672: Job 01000000 already started in worker pool.
[2023-09-04 15:22:02,680 I 167 167] (raylet) agent_manager.cc:40: HandleRegisterAgent, ip: ***.**.*.*, port: 57617, id: 424238335
[2023-09-04 15:22:04,165 I 167 205] (raylet) agent_manager.cc:131: Agent process with id 424238335 exited, exit code 0. ip ***.**.*.*. id 424238335
[2023-09-04 15:22:04,165 E 167 205] (raylet) agent_manager.cc:135: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. Agent can fail when
- The version of `grpcio` doesn't follow Ray's requirement. Agent can segfault with the incorrect `grpcio` version. Check the grpcio version `pip freeze | grep grpcio`.
- The agent failed to start because of unexpected error or port conflict. Read the log `cat /tmp/ray/
grpcio version is 1.57.0.
Please suggest if there are any work arounds.