How severe does this issue affect your experience of using Ray?
I have a local on-premise cluster of 5 identical machines (8 cores each).
I would like to restrict ray to only use 4 (of the 8) cores on the head node.
How can I do this?
cc @Chen_Shen, do you have insight here?
hi @xydote Ray doesn’t have built-in support using a proportion of physical resources on a single machine (with the exception of GPUs).
There is some ongoing work to use Ray workers in containers (where you can limit the resource used by the container) but it remains experimental.
Perhaps I am not understanding the question but couldn’t they just start ray on that node with
ray start / ray.init with the argument num_cpus=4?
@mannyv that’s a great question.
So when you override those system resources, it only affects ray’s scheduler (so it won’t schedule more than 4 tasks if the node is allocated with 4cpus) ; but it doesn’t prevent your task (i.e. if you started multiple threads in a single task) to use more resources than needed.
https://docs.ray.io/en/master/ray-core/tasks/resources.html
If specifying CPUs, Ray does not enforce isolation (i.e., your task is expected to honor its request).
Yeah this is not possible at the Ray level. If you’d like to achieve it, you can probably deploy ray within a container (with CPU restriction on a container).