How to prevent scheduling non-GPU tasks to GPU nodes

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

Context: I have some tasks that require a GPU node, which is started on demand using kuberay. Once that task finishes, another task which doesn’t require a GPU starts and it’s scheduled to the currently active GPU node. Ideally, I would like this task to spin-up a high-memory non-GPU node, allowing the GPU node to be deallocated. However, Ray prefers to schedule the task to the already available GPU node before it spins down.

Is there a way to define an affinity for task that prevent them to be scheduled to GPU nodes? I know it’s possible to define the specific node_id for a task. But that is not what I need, since the node_id is not known a priori, as the nodes scale up and down automatically.

Why: The main reason to do this is saving compute resources/money.

@eduardoarnold see Placement Groups: Placement Groups — Ray 2.31.0