As I describe in another question, I’m sending work to hundreds of Ray Actors using a Queue. These Ray Actors are distributed across several Ray Worker nodes. How does Ray decide what actors to send to which nodes?
I ask, because currently two of my nodes are being given almost all the Actors (according to the Ray dashboard) and being used at max capacity, while my third is sitting idle. This can be seen in the Graphana plot below, where each colour demonstrates the CPU usage of each of my Workers. The Yellow and Green workers are being used extensively, while Blue mostly sits idle.
If Ray naively balanced the load by equally allocating the same number of Actors across all nodes, I wouldn’t be seeing such unequal usage. Given my problem, I’m assuming Ray is using some other method?
What method is it using for distributing Actors to Workers and can I change it?