`get_gpu_ids` is not empty but `torch.cuss.is_available` is false

Ray Core
1

Hi,

I am new to ray and while trying to use it I faced a problem that my googling abilities failed to help me with.

The setup is the following. I am launching ray on a local machine in Docker initializing it with a single GPU and ten CPU cores.

One worker fails with

RuntimeError: No CUDA GPUs are available

having get_gpu_ids equals to [0] but torch.cuda.is_available() is false. Whereas there is one process with torch.cuda.is_available() equals to true.

Is this expected behavior? If yes, could please point to the manual so that I can more fully understand how ray works? If no, what can I do to avoid this error?

Launching on Ubuntu 18.04, PyTorch of version 1.9+cu111, ray of version 1.8.

Thanks.

2

cc @rliaw @sven1977 can you guys address this question?