How to use a particular node for ray serve?

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I have a ray cluster on AWS with multiple node types like V100s, A10Gs, etc.
I want to understand a way in which I can define ray serve to use a particular worker node. For example, if the name of the worker node is “my_a10g_node”, how can I use it? I looked through the custom resource but didn’t understand how to use it with Ray serve.

Any examples of usage will be helpful.

Hi @plum9, does using ray_actor_options work, together with custom resources? Scaling and Resource Allocation — Ray 2.0.0

So you could run ray start --resources='{"a10g": 1}' on your special node when starting Ray, and then pass in the resource requirement to Serve deployment as follows:

@serve.deployment(ray_actor_options={"resources": {"a10g": 0.1}})
def func_1(*args):
    return do_something_with_my_a10g_node()