Synchronous/Sequential scheduler with ray tune

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hello,

I am a newcomer to ray tune and I until now I was using the ASHAScheduler for hyperparameter tuning.

It was working well as long as I used small networks but now that I am trying to tune bigger networks the program runs out of memory as I only have two GPUs available.

More precisely, one big network takes both GPUs memory so when the program runs two of them in parallel the second one runs out of memory.

My question is: does a sequential scheduler exist in ray tune ? So that I can run each trial one at a time, even if I am aware that it will take more time.

1 Like
2

@tim is it possible to enforce the first network to only run 1 GPU at a time?

Alternatively, you can use local mode to force sequential execution - https://docs.ray.io/en/master/tune/user-guide.html#debugging

3

Hi, thank you for your answer. I will try it :+1: