How does Ray load-balance Actors across Ray Workers?

Ray Clusters
1

As I describe in another question, I’m sending work to hundreds of Ray Actors using a Queue. These Ray Actors are distributed across several Ray Worker nodes. How does Ray decide what actors to send to which nodes?

I ask, because currently two of my nodes are being given almost all the Actors (according to the Ray dashboard) and being used at max capacity, while my third is sitting idle. This can be seen in the Graphana plot below, where each colour demonstrates the CPU usage of each of my Workers. The Yellow and Green workers are being used extensively, while Blue mostly sits idle.

image
image3626×422 49 KB

If Ray naively balanced the load by equally allocating the same number of Actors across all nodes, I wouldn’t be seeing such unequal usage. Given my problem, I’m assuming Ray is using some other method?

What method is it using for distributing Actors to Workers and can I change it?

2

Ray and others systems usually try to schedule tasks locally, to avoid the cost of moving data. Said that you can specify resources for your actors (number of cpus/gpus, memory or a custom resource) or use the placement group API. Using these resources would spread more evenly the actors across your cluster.