How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
How many worker ports should the head node have open for a custom cluster? I am currently opening 3x as many ports as the number of CPUs per host via the worker-port-list option (for both the head and worker nodes) but am encountering the following warning message:
WARNING: 80 PYTHON worker processes have been started on node: <head node> with address <head node ip>. This could be a result of using a large number of actors, or due to tasks blocked in ray.get() calls (see https://github.com/ray-project/ray/issues/3644)
The task I ran has a dummy objective function that sleeps for 1 to 10 seconds, and I am using Ray Tune to do a sweep of 10k samples. The issue I’m running into is that this Tune run sometimes completes without issue while hanging indefinitely at other times.
Does anyone have any insight on what causes this warning to be emitted? Thanks.