How to execute each experiment once?

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hi,
I would like to execute each possible experiment from a list exactly once.

Running the MWE following, experiments with an “exp” value of 0,1,2,3,4 should be done.
However, tune randomly picks numbers from the “exp” list and repeats them.

What am I doing wrong?

from ray import tune
from functools import partial

config = {
    "exp": tune.choice([0,1,2,3,4]), 
}


def dummy(config, checkpoint_dir=None, data_dir=None, workers=5):    
    print("Experiment: ",config["exp"])
    
    
def main(trainable, num_samples=10, max_num_epochs=10, gpus_per_trial=0.5, cpus_per_trial=10, name="testrun"):
    
    global result
    result = tune.run(partial(tune.with_parameters(trainable, workers=cpus_per_trial)),
                      config=config,
                      num_samples=num_samples,
                      name=name,
                      verbose=True,
                      local_dir="/ray_results"
                     )
   
if __name__ == "__main__":
     main(trainable=dummy,num_samples=5, max_num_epochs=1, name="dummy")
     # This should execute each experiment (0 to 4) once, but it randomly executes experiments with replacement.
2

For your example, you could try:

config = {
    "exp": tune.grid_search([0,1,2,3,4]), 
}
...
...

if __name__ == "__main__":
     # we set num_samples = 1 here because it will repeat the grid.
     main(trainable=dummy, num_samples=1, max_num_epochs=1, name="dummy")
3

Thank you very much, this is great!