This is an incredibly important functionality and has been asked before, but I did not see a satisfactory solution.
If I have N hyperparameter settings in my search space, I want to repeat each of these with k different random seeds, and report the aggregate (mean, or worst) metric over these seeds.
repeater has been suggested for this, but there is a caution not to use it with a Trial Scheduler, e.g. those with early termination. Well… early termination is super-useful, so is there some way to get both early termination, and the above functionality?
Also, it would be helpful if someone could clarify what precisely is the problem with early-termination when using