I ran a hyper-parameter tuning experiment with a few discrete parameters that get selected using tune.choice for a total of 24 total parameters. I then set the num_samples parameter in tune.Tuneconfig precisely to 24.
I was expecting ray.tune just to do a grid search across all possible parameters, but he somehow retries different parameters multiple times. Is this the expected outcome or there has to be an error somewhere?
Yes, this is expected. Using tune.choice with num_samples=24 will randomly sample 24 parameter combinations, possibly with repeats, not exhaustively enumerate all unique combinations. For a true grid search (i.e., every possible combination exactly once), you must use tune.grid_search for each parameter instead of tune.choice—this will ensure all combinations are covered without repetition, regardless of num_samples [source], [source], [source].
Would you like a code example or more detail on how to set up grid search in Ray Tune?
Sources:
Hint: Mention @RunLLM in the post for followups.