I took another look at our documentation with code. I believe that user could set CUDA_VISIBLE_DEVICE through env var. And if that is set, Ray will respect that in the sense that only GPUs in this list will be returned.
See worker.py - ray-project/ray - Sourcegraph
Hi @lesolorzanov, that should work though (we use it all the time in our end to end testing). Just out of curiosity, are you using a grid search? Because num_samples=1 means you’re only going to start 1 sample otherwise.
If that’s the case, it seems that it’s stil lscheduling on the wrong GPU. Can you confirm with nvidia-smi where the trials are scheduled exactly?