My dashboard says GPU [0] NA

Ray Clusters Kubernetes
1

GPU
[0]:N/A
GRAM
[0]: 0 MiB / 16160 MiB. ---------> ray dashboard

the cluster thinks there is one gpu because it can schedule this job
but inside the function when torch wants to use gpu, it cannot find any gpu

e_gpu pid=729) ray.get_gpu_ids(): [0]
(use_gpu pid=729) CUDA_VISIBLE_DEVICES: 0

why its showing like this i have set gpu = 1

2

@matthewdeng , do you think you can help?

3

@subbu_prasant were you able to figure this out?

ray.get_gpu_ids(): [0] and CUDA_VISIBLE_DEVICES: 0 are indicating that the task is associated with GPU with ID 0.

For more details, see GPU Support — Ray 1.12.0