# \[tune\] multiple runs with same hyperparameter, different random seed

**URL:** <https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644>\
**Category:** Ray Tune\
**Created:** [January 28, 2021, 8:01pm UTC](https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644 "2021-01-28T20:01:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![avandekleut](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/avandekleut/32/350_2.png) [@avandekleut](https://discuss.ray.io/u/avandekleut)\
**Post date:** [January 28, 2021, 8:01pm UTC](https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644/1 "2021-01-28T20:01:47Z")

</div>

I want to re-run the same experiment multiple times with the same hyperparameters and different random seeds, so I can average the results over multiple runs. How can I do this? I can’t use a Repeater because that requires a Search algorithm (which I don’t need, I’m just randomly sampling 40 hyperparameter sets from the search space).

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 28, 2021, 8:12pm UTC](https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644/2 "2021-01-28T20:12:25Z")

</div>

Maybe try `tune.run(config={"seed": tune.grid_search([1, 2, 3, 4,5])})`

You’ll have to average the results manually though.

---

<div class="post-metadata">

**Author:** ![avandekleut](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/avandekleut/32/350_2.png) [@avandekleut](https://discuss.ray.io/u/avandekleut)\
**Post date:** [January 28, 2021, 11:53pm UTC](https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644/3 "2021-01-28T23:53:08Z")

</div>

I want tune to generate the other hyperparameters for me. Image

```auto
from ray import tune

config = dict(
    feature_dim=tune.choice([128, 256]),
    lr=tune.choice([0.001, 0.0001]),
    seed=tune.grid_search([1, 2, 3, 4, 5])
)

def trainable(config):
    print(config)
    tune.report(score=0)

tune.run(trainable, config=config, num_samples=10)

```

When running this, while `seed` is grid-searched over, the other hyperparameters are sampled randomly. Instead, I would want the same hyperparameters repeated 5 times except for the seed which changes. I would want to generate 10 hyperparameter sets (except for the seed), resulting in 50 runs in total. I can handle the averaging myself post-hoc.

---

<div class="post-metadata">

**Author:** ![avandekleut](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/avandekleut/32/350_2.png) [@avandekleut](https://discuss.ray.io/u/avandekleut)\
**Post date:** [January 29, 2021, 12:29am UTC](https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644/4 "2021-01-29T00:29:17Z")

</div>

Also, the `Repeater` search won’t work for me because I report two different metrics at different points, and no search algorithm that can be wrapped by repeater works with that setup… And I don’t understand how to subclass the `Searcher` class to generate configurations (not well documented and code for the search algorithms changes a lot).

---

<div class="post-metadata">

**Author:** ![avandekleut](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/avandekleut/32/350_2.png) [@avandekleut](https://discuss.ray.io/u/avandekleut)\
**Post date:** [January 29, 2021, 1:01am UTC](https://discuss.ray.io/t/tune-multiple-runs-with-same-hyperparameter-different-random-seed/644/5 "2021-01-29T01:01:55Z")

</div>

Alright, after a lot of thinking I ended up deciding that, for my use case, I can generate the hyperparameter variants myself and ask ray to run them in parallel by just using a nested config with a grid search. I just do a grid search over all the generated variants to let ray handle parallelism, gpu assignment, etc.

```python
from ray import tune
import copy

def generate_variants(config, num_seeds=1, num_samples=1, seed_generator = lambda seed: seed):
    """
    Requires all elements to have a .sample method, like tune.choice etc. 
    cannot be grid_search.
    Given a config of hyperparameters, generates num_samples of them by 
    calling .sample on each hyperparameter seach space. Then makes 
    num_seeds copies of these, each with a different random seed.
    """
    variants = []
    for sample in range(num_samples):
        variant = {param:space.sample() for param, space in config.items()}
        for seed in range(num_seeds):
            seeded_variant = copy.deepcopy(variant)
            seeded_variant['seed'] = seed_generator(seed)
            variants.append(seeded_variant)
    return tune.grid_search(variants)

hyperparams = dict(
    feature_dim=tune.choice([128, 256]),
    lr=tune.choice([0.001, 0.0001]),
)

config = generate_variants(hyperparams, num_seeds=3, num_samples=2)

def trainable(config):
    tune.report(**config)

tune.run(trainable, config=config)

```