How to update a bundle in a placement group

Ray Core
1

I need to create a placement group with a bundle: {“CPU”:32}, where 32 is the total available cpus on a node in my cluster. So basically I need to reserve this node for some of the tasks.
Then in order to schedule the tasks, I also need to pass in the nodeid, e.g., node:127.0.0.1 to the options. so that, the actual resource requirement is

{'node:172.31.45.197': 0.0001, 'CPU': 0.9}

but the bundle I have in the pg is

{'CPU':32}

Ray scheduler fails to figure out this matching, so I wonder how can I update the bundle with the node_id info? such that after creating the pg, I have

{'CPU':32,'node:127.0.0.1':1}

I have a workaround: maybe I can just get all node id first, then create the pg, this however, I think, won’t work in an auto-scalable cluster.

Any thoughts?

3

I don’t understand why task’s request has to be the bundle granularity. It looks like, I can schedule multiple concurrent tasks requesting {‘CPU’:0.9} to fit into the bundle {‘CPU’:32}.

4

Ah, sorry, please ignore my earlier message, I was mistaken about the semantics for scheduling tasks inside a placement group bundle. You are right, it is fine to schedule a task with {“CPU”: 0.9} into the bundle {“CPU”: 32}.

For your original question, can I ask why you need to schedule with the node ID? Once the placement group is scheduled, you should not need the node ID to place the task on that node anymore.

5

the reason is the pg I created has multiple nodes, and I think I’d better load-balance the tasks on those nodes.

6

Ah I see, makes sense! Yes, unfortunately you are also right that there isn’t a good way to do this at the moment… I think you will have to either:

  1. Rely on Ray’s default policy to load-balance tasks within the placement group - this should be good enough for most cases, but please do let us know if you run into issues with this.
  2. Create one placement group for each node and manually load-balance tasks between the placement groups.