Autoscaling vertex ray not going to min_replica=0

Ray Clusters
1

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.

2. Environment:

  • Ray version: 2.47
  • Python version: 3.12
  • OS:
  • Cloud/Infrastructure: GCp-vertex ray agent
  • Other libs/tools (if relevant):

3. What happened vs. what you expected:

  • Expected: Autoscale to 0 when min replica count set with ray
  • Actual: 1 gpu always alive with Head node.

I ran a pytorch matmul.cuda() on a ray cluster in vertex ray agent platform with no ray and its scale on the same cluster to 0 but when i run the same thing with Ray (torch matmul) the gpu doesnt scaled fully to 0 for the worker pool. there is this dashboard or stats actor always running with gpu live for days. Strange behaviour.

2

Ray clusters typically keep the head node running, and some system actors (like stats or dashboard actors) may remain alive and can be scheduled on GPU nodes, preventing full GPU scale-down even when min replicas are set to 0. This is a known behavior, especially if system actors are not restricted to CPU-only nodes, causing GPU nodes to stay alive unnecessarily after user jobs finish. Manual intervention (e.g., killing lingering actors) is sometimes needed to release GPU resources completely, as discussed in community threads and code examples (How does Ray actor work?).

To avoid this, ensure system actors are scheduled on CPU nodes or configure the head node to not claim GPU resources. In Kubernetes, set num-gpus: "0" for the head node in your RayCluster spec to prevent the head from reserving GPUs (Troubleshoot multi-node GPU serving on KubeRay). Would you like more detail on how to configure this or how to identify and kill lingering actors?

Sources:

Hint: Mention @Herald in the post for followups.

3

@RunLLM
@Herald
Since I am using vertex-ray image on the ai agent platform to develop ray application:

BASE IMAGE: FROM europe-docker.pkg.dev/vertex-ai/training/ray-gpu.2-47.py311:latest

How can I then control or ensure actors to be on the cpu nodes and not claim GPU nodes in the below code example. I can test the case to see on my cluster. Convert this code example so I test it out

Current cluster setup:
Accelerator: NVIDIA_TESLA_T4x1
Head node_count=1
Scaling replica count is: min=0 max=2

#!/usr/bin/env python3
"""Run N PyTorch GPU matmuls in parallel as Ray tasks, then shut down cleanly.

Starts Ray, fans out NUM_WORKERS matmul tasks (each requesting num_gpus=1, so the
autoscaler adds up to NUM_WORKERS GPU workers), frees GPU memory on every worker,
gathers the results, then shuts Ray down gracefully. Run directly on the cluster
or submit as a job entrypoint:

    python data/ray_test/matmul_ray.py

Set NUM_WORKERS / MATMUL_SIZE via env vars to tune fan-out and matrix size.
"""

from __future__ import annotations

import gc
import os

import ray
import torch

NUM_WORKERS = int(os.getenv("NUM_WORKERS", "4"))
MATMUL_SIZE = int(os.getenv("MATMUL_SIZE", "4096"))


@ray.remote(max_calls=1, num_gpus=1)
def matmul(worker_id: int, size: int) -> dict:
    if not torch.cuda.is_available():
        raise RuntimeError(f"[worker {worker_id}] CUDA not available on the Ray worker")

    device = torch.device("cuda")

    @torch.compile
    def mm(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        return x @ y

    a = torch.randn(size, size, device=device)
    b = torch.randn(size, size, device=device)
    c = mm(a, b)  # first call triggers compilation (Inductor), then runs
    torch.cuda.synchronize()
    result = {
        "worker_id": worker_id,
        "node": ray.get_runtime_context().get_node_id(),
        "device": torch.cuda.get_device_name(0),
        "torch": torch.__version__,
        "trace": float(c.trace().item()),
        "peak_mib": torch.cuda.max_memory_allocated() / 1024**2,
    }

    # Clean cache: drop refs -> gc.collect() so refcounts hit zero -> empty_cache().
    del a, b, c
    gc.collect()
    torch.cuda.empty_cache()
    result["alloc_after_mib"] = torch.cuda.memory_allocated() / 1024**2
    return result


def main() -> None:
    ray.init(address="auto")
    try:
        print(f"Fanning out {NUM_WORKERS} matmul tasks (size={MATMUL_SIZE})")
        refs = [matmul.remote(i, MATMUL_SIZE) for i in range(NUM_WORKERS)]
        for res in ray.get(refs):
            print(
                f"  worker {res['worker_id']} on node {res['node'][:12]} "
                f"({res['device']}) trace={res['trace']:.4f}",
            )
        # Drop refs so workers hold no objects -> nodes go idle and can scale down.
        del refs
        gc.collect()
    finally:
        ray.shutdown(_exiting_interpreter=True)
        print("ray.shutdown() complete — driver disconnected")


if __name__ == "__main__":
    main()