# How to integrate TorchTrainer with Ray Tune and HyperOpt space?

**URL:** <https://discuss.ray.io/t/how-to-integrate-torchtrainer-with-ray-tune-and-hyperopt-space/9392>\
**Category:** Uncategorized\
**Created:** [February 15, 2023, 11:31pm UTC](https://discuss.ray.io/t/how-to-integrate-torchtrainer-with-ray-tune-and-hyperopt-space/9392 "2023-02-15T23:31:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mnazemi](https://avatars.discourse-cdn.com/v4/letter/m/9dc877/32.png) [@mnazemi](https://discuss.ray.io/u/mnazemi)\
**Post date:** [February 15, 2023, 11:31pm UTC](https://discuss.ray.io/t/how-to-integrate-torchtrainer-with-ray-tune-and-hyperopt-space/9392/1 "2023-02-15T23:31:27Z")

</div>

`TorchTrainer` expects the tuning configuration to be provided in the `{"train_loop_config": config}` format as shown in [this example](https://docs.ray.io/en/latest/train/dl_guide.html#hyperparameter-tuning-ray-tune).

However, when using custom search spaces, e.g., one that is defined with `hyperopt`, the `config` is automatically generated by Ray and passed to the trainer function, so it does not conform to the `{"train_loop_config": config}` format.

A quick and dirty solution to this problem is adding something like `self.config = {"train_loop_config": self.config}` before the dictionaries are merged [here](https://github.com/ray-project/ray/blob/f6181840068d10d2eeb85ff43c9f332b0e59d450/python/ray/train/base_trainer.py#L428).

Is there a cleaner way to pass configurations generated from spaces to `TorchTrainer`?

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [February 16, 2023, 5:43pm UTC](https://discuss.ray.io/t/how-to-integrate-torchtrainer-with-ray-tune-and-hyperopt-space/9392/2 "2023-02-16T17:43:57Z")

</div>

Do `param_space={"train_loop_config": {"a": tune.grid_search([1, 2])}}` and `Tuner(..., searcher_alg=HyperOptSearch())` work for you?

---

<div class="post-metadata">

**Author:** ![mnazemi](https://avatars.discourse-cdn.com/v4/letter/m/9dc877/32.png) [@mnazemi](https://discuss.ray.io/u/mnazemi)\
**Post date:** [February 16, 2023, 7:13pm UTC](https://discuss.ray.io/t/how-to-integrate-torchtrainer-with-ray-tune-and-hyperopt-space/9392/3 "2023-02-16T19:13:28Z")

</div>

It will probably work. I was just curious to see if it could be done automatically without changing the `param_space`.

The way I currently use custom `hyperopt` spaces does not work with Tune out of the box. For example, assume the following conditional search space:

```auto
space = {
    ##### Augmentation ######
    # Cutmix
    "cutmix": hp.choice(
        "ctmx",
        [
            {
                "cutmix": True,
                "cutmix_beta": 1.0,
                "cutmix_prob": hp.quniform("cutmix_prob", 0.10, 0.50, 0.05),
            },
            {"cutmix": False},
        ],
    ),
}

```

The configuration that Tune passes to workers looks like this:

```auto
{"cutmix": {"cutmix": False}}

```

If your code is designed to receive `cutmix`, `cutmix_beta` and `cutmix_prob`, the provided config will not work, i.e., you need to process spaces defined in Tune, which only have the expected keys, and the ones defined with `hyperopt`, which may have nested dictionaries, differently.

So I have written a helper function to first flatten the nested dictionaries, and use the updated `config` in the worker.

```auto
def hyperopt_to_ray(config):
    # Flatten nested config when HyperOpt custom space is defined
    outer_keys = []
    for key in config:
        if isinstance(config[key], dict):
            outer_keys.append(key)
    for key in outer_keys:
        inner_dict = config.pop(key)
        config.update(inner_dict)

```

Changing the `param_space` definition to something like `param_space = {"train_loop_config": param_space}` would make this whole process of dealing with spaces more complicated.