Hi, as a newbie to Ray Tune, I am wondering if I can run each hyper-parameter configuration multiple times with different data-split seeds, and choose the configuration with the best average performance over seeds? It seems I can naively achieve this by analyzing the result dataframe myself, but I’m wondering if there is a built-in mechanism for automatically grouping results based on seeds?
You should be able to use something like the Repeater tool. Search Algorithms (tune.suggest) — Ray v2.0.0.dev0