Ray ignores first gpu in multi-gpu environment?

1

I have a local server with 8 GPUs.
No matter the configuration I’m using, ray is always ignoring the first GPU
ray.init(local_mode=False, num_cpus=10, num_gpus=8)

image
image3010×734 377 KB

Ideas?

2

Ray scheduler doesn’t assign the GPU to each worker, but it is just for the scheduling. My guess is your application code is not properly using the first GPU for some reason. Are you using Tune or Rllib?

3

Thanks for the response. I’m using tune. When running in local mode Ray uses the first gpu. When I’m not using Ray at all the first gpu is chosen and runs properly.

4

cc @rliaw Do you know what’s the cause of this issue?

5

No, I don’t unfortunately. Maybe you can look at the dashboard to see what’s going on?