I’m trying to resume the tuning after expanding the search space by adding more hyperparameters. The tune won’t run, I guess, due to the mismatch between the previous and current search space. Is there a way to continue the training in this case?

Hi, This is not supported now. Would it work if you just start a new tune run with each trial loading from previously last checkpoint?

resume tuning works nicely without changing search space. Is it possible to manually update some configuration files so that it can work with the modified search space?

It is currently not supported. Have you tried my last suggestion? You can start another tune run with new search space and each trial can either start from fresh or start from a loaded checkpoint (if the hyperparameter combination is ever in last tune run).

Can you please be a bit more specific? Currently I use algorithm.restore_from_dir() to resume the run. Is that what you are suggesting?

What algorithm do you use? Does it support resuming with updated hyperparameter space? Beyond the limitation of specific search algorithm, I don’t think ray tune supports that well either. However, if you just want to continue tuning with a say selected sub hyperparameter space, without wasting all…

Thanks. I’m using hyperopt. I’m assuming it read the checkpoint using the function restore_from_dir() which is equivalent to “get_latest_checkpoint_with_hyparam” function. Is that a correct understanding?

I think get_latest_checkpoint_with_hyperparam is just so that to not waste the existing effort accumulated on a particular hyperparam combination. If you are not concerned about that, and just want to resume searcher state, can you try something like the following in the 2nd run? hyperopt_searcher…

That’s actually what I did for a different issue in the following thread: [image] Correct way of using tuner.restore() Ray Tune According to the source code comments, the restore() function is only for “”"“Restores Tuner after a previously failed run.”. If a previous run …

Does that solve the issue for you?

Nope. This is actually not the right way to continue a finished tune. It can only work on the tune that’s not finished. The conclusion from that thread is to use algorithm.restore_from_dir() to continue a finished tune which I believe is the same as `get_latest_checkpoint_with_hyperparam(). In the …

Resume tuning after updating search space with more hyperparameters

Uncategorized

wxie2013 February 14, 2023, 8:04pm 9

That’s actually what I did for a different issue in the following thread:

Topic		Replies	Views
Continue the tuning after changing the searching range Uncategorized	3	388	June 26, 2023
Correct way of using tuner.restore() Ray Tune	6	2459	November 16, 2022
Resuming tune optimization from previously explored configurations Uncategorized	2	1182	October 3, 2023
Continue training for successful ray tune candidates Uncategorized	3	1032	October 7, 2022
Add additional trials to experiment Ray Tune	5	970	May 3, 2025

Resume tuning after updating search space with more hyperparameters

Related topics