Task assignment to multiple workers in autoscaler

Ray Clusters Kubernetes
1

Hello,

Do we log tasks executed by workers in ray when the cluster is scaled, a user might want to know if all the workers are busy running tasks to tune max_workers when running a workload.

Thanks

2

@sangcho could have some insights here.

3

This is currently not tracked by Ray. One thing you can try is to use one actor as a metric collector and report it to the actor whenever task is executed. There’s no immidiate plan I am aware of that we will add soon to support this. This StatusTrakcer in shuffle.py code is kind of an example; ray/shuffle.py at 0f921cdead7c4752ce5ee87b8a1d33ad1cb3a004 · ray-project/ray · GitHub (for this case, whenever task finishes creating objects, it reports to the StatusTrakcer actor, and in the driver, we render a progress bar)

4

Thanks! learned something new. Maybe this is a stretch, do you know how to get node names or IP addresses executing tasks?

5

You can use this API to obtain “node_id”. Node id is an id for each raylet (note that there can be multiple node ids in the same node ip if raylets are killed and restarted by autosclaer). API and Package Reference — Ray v2.0.0.dev0

If you’d like to get node ip, you need to use our private API path (which is not recommended). If you need this feature, please file a feature request with use case examples!