Cluster not being utilized

Ray Core
1

I have a local cluster, 19 nodes including the head, with 436 total workers… I’ve confirmed they are alive and available… I have created a large queue of tasks, via a list of calls to some function.remote, and then am processing the results via ray.wait … when looking the dashboard, it is only showing 76/436 being utilized… and really only 4 of 19 nodes… the rest are all showing 0/24 … I’m on Ray 1.2 … the function iteself is chunking a .csv file and returning some match if available. Any suggestions to troubleshoot?

2

Ray 1.2 (and until 1.3), Ray prefers to saturate nodes one by one. That says tasks won’t likely to go other nodes until the first N nodes are fully saturated.

At 1.3, you can change the behavior to randomly load balance tasks to all nodes in the cluster by starting a head node with system-config


ray start --head --system-config='{"scheduler_loadbalance_spillback":true}'
3

Note 1.3 will be released in about 2~ish weeks.

4

thanks for that update… but if the tasks are significantly greater than the number of workers, i.e. I submitted like 240k tasks to 436 workers… why does the dashboard seem to only show PID for 76-100 out of 436 workers … when I took a different tact… and instead moved the task to an Actor, and then created an ActorPool of ~400 workers… I then see the cluster being saturated with work on the dashboard… I’m just wondering is it just a dashboard thing… when submitting tasks, is it really not utilizing the cluster, or is it just a reporting issue?

5

Hmm I see. I have a couple questions;

  • How long are your tasks?
  • Can you also try running ray status and see if the number of nodes are all correct?