This is an incredibly important functionality and has been asked before, but I did not see a satisfactory solution.
If I have N hyperparameter settings in my search space, I want to repeat each of these with k different random seeds, and report the aggregate (mean, or worst) metric over these seeds.
I know repeater has been suggested for this, but there is a caution not to use it with a Trial Scheduler, e.g. those with early termination. Well… early termination is super-useful, so is there some way to get both early termination, and the above functionality?
Also, it would be helpful if someone could clarify what precisely is the problem with early-termination when using repeater.
Thanks
Hello!
I am not an expert on the subject, but the problem with making an early stop is that then the average value for a hyperparameter configuration that was executed repeatedly with different seeds can be distorted from the value it would have reached on average, if these are executed every once completely.
In any case, I still don’t understand when the average is done. Is it done in each new repetition, that is, the last repetition of each hyperparameter configuration is the one that gives us the final average? Or is the averaged value delivered at the end of the run, when all configurations are done?
I have not been able to observe the report of the average value of each configuration.
Have you been able to advance in your understanding of this tool? Anyone who can help to better understand how to analyze the results obtained with the repeater?