[tune] multiple runs with same hyperparameter, different random seed

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

I want to re-run the same experiment multiple times with the same hyperparameters and different random seeds, so I can average the results over multiple runs. How can I do this? I can’t use a Repeater because that requires a Search algorithm (which I don’t need, I’m just randomly sampling 40 hyperparameter sets from the search space).

2

Maybe try tune.run(config={"seed": tune.grid_search([1, 2, 3, 4,5])})

You’ll have to average the results manually though.

3

I want tune to generate the other hyperparameters for me. Image

from ray import tune

config = dict(
    feature_dim=tune.choice([128, 256]),
    lr=tune.choice([0.001, 0.0001]),
    seed=tune.grid_search([1, 2, 3, 4, 5])
)

def trainable(config):
    print(config)
    tune.report(score=0)

tune.run(trainable, config=config, num_samples=10)

When running this, while seed is grid-searched over, the other hyperparameters are sampled randomly. Instead, I would want the same hyperparameters repeated 5 times except for the seed which changes. I would want to generate 10 hyperparameter sets (except for the seed), resulting in 50 runs in total. I can handle the averaging myself post-hoc.

4

Also, the Repeater search won’t work for me because I report two different metrics at different points, and no search algorithm that can be wrapped by repeater works with that setup… And I don’t understand how to subclass the Searcher class to generate configurations (not well documented and code for the search algorithms changes a lot).

5

Alright, after a lot of thinking I ended up deciding that, for my use case, I can generate the hyperparameter variants myself and ask ray to run them in parallel by just using a nested config with a grid search. I just do a grid search over all the generated variants to let ray handle parallelism, gpu assignment, etc.

from ray import tune
import copy

def generate_variants(config, num_seeds=1, num_samples=1, seed_generator = lambda seed: seed):
    """
    Requires all elements to have a .sample method, like tune.choice etc. 
    cannot be grid_search.
    Given a config of hyperparameters, generates num_samples of them by 
    calling .sample on each hyperparameter seach space. Then makes 
    num_seeds copies of these, each with a different random seed.
    """
    variants = []
    for sample in range(num_samples):
        variant = {param:space.sample() for param, space in config.items()}
        for seed in range(num_seeds):
            seeded_variant = copy.deepcopy(variant)
            seeded_variant['seed'] = seed_generator(seed)
            variants.append(seeded_variant)
    return tune.grid_search(variants)

hyperparams = dict(
    feature_dim=tune.choice([128, 256]),
    lr=tune.choice([0.001, 0.0001]),
)

config = generate_variants(hyperparams, num_seeds=3, num_samples=2)

def trainable(config):
    tune.report(**config)

tune.run(trainable, config=config)
1 Like