Ray Worker Max Memory

Ray Core
1

Is there a way so specify the maximum amount of memory available to a ray worker, same as you do with the --num-cpus option?

2

So there are two things I’d like to answer here.

  1. num-cpus doesn’t guarantee the resource isolation. It is purely used for scheduling bookkeeping. That says, although you specify num_cpus=0, it is still possible to use a cpu.
  2. Currently, there’s no way to add memory restriction for "“worker”. but you can do this for task or actor using memory argument in ray.remote. For actors, we are trying to restrict he memory usage at best effort (meaning actor will crash if it uses more m emory than your allocation).

If adding memory restriction to each worker is important for your use case, please file a feature request to Ray github page! (with detailed use case). Issues · ray-project/ray · GitHub

3

Thank you sangcho for your response.

I understand both your points and i already know that there is no isolation. However, the num_cpus you define for a worker plays a part when placing actors and tasks that declare resource needs, so my point was if there is a way to define the amount of memory for the same purpose.

In my case i’m using k8s, and i’m not able to understand where the values for memory shown when calling ray.cluster_resources() and in ray.nodes() come from. Did i miss an explanation of this somewhere in the documentation?

Thanks

4

I think you’re referring to the weird units (memory is shown in 50MiB chunks due to some limitations in our legacy code). To convert to bytes, you can do ray.cluster_resources["memory"] * ray.ray_constants.MEMORY_RESOURCE_UNIT_BYTES.

1 Like