Too many pyhton processes on Node

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

HI, I am using ray cluster and the basic situation is as follows.

Debian 11;
python3.9.16;
ray 2.2.0;
1 head and 1 node;

If I submit ray tasks to the cluster many times, many python3.9 processes will appear on the node, and it seems to be increasing, I am worried that this will affect the performance of the node or the submission of ray tasks when executing many tasks, is there any good way to solve this.
Thanks!

1
11230×574 87.8 KB

2

Are you running from a local machine or a container? I found that with some orchestrator like OpenShift, when the CPU bursting is turned on, Ray does not detect the CPU available correctly.

Suggest you just set --num-cpus in ray start (or the analogous parameter for ray init if you are just starting Ray from Python) and the number of processes should be limited.

1 Like
4

Thanks for your reply and suggestion!