ASHA scheduler stops runs just after the grace period even though there are no other runs?

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hello,

I’m running hyperparameter with Ray Tune and I am using the Optuna search algorithm in combination with the ASHA scheduler.

My problem is that runs that reaches just after the grace period are stopped, even though there are no other runs. That did not happen with the first I ran a hyperparameter search. I suspect that it uses some metadata from previous searches? Is it possible to programmatically start hyperparameter fresh, that is always with a clean state?

Thank you

2

Hi @lthiet, it would be helpful if you could provide the script that you’re running.

Does the log indicate that your experiment is being resumed? It should only be the case if you specify resume="AUTO" to tune.run() or if you use Tuner.restore().

Anyway, you can avoid using the same directory by passing a unique name to your RunConfig. If you don’t pass a name at all, it will default to a unique name so you shouldn’t run into problems.