This morning I dug into this a bit and found that by setting:
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
ray.init(num_gpus=1)
the issue was resolved (in my use case, although I suspect in the example in which I posted the results the behavior would be similar). This was after confirming via the normal procedure that cuda was available and the device id was 0. It feels like there is an issue with the GPU being detected automatically and thus once we get to ray.run()
the gpu settings in the config
parameter have no effect.
Not sure if this is WSL 2 or Docker specific.