# Tune custom gym env\_config params with PBT

**URL:** <https://discuss.ray.io/t/tune-custom-gym-env-config-params-with-pbt/10673>\
**Category:** Ray Tune\
**Created:** [May 16, 2023, 7:53am UTC](https://discuss.ray.io/t/tune-custom-gym-env-config-params-with-pbt/10673 "2023-05-16T07:53:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![PREJAN](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/prejan/32/4518_2.png) [@PREJAN](https://discuss.ray.io/u/PREJAN)\
**Post date:** [May 16, 2023, 7:53am UTC](https://discuss.ray.io/t/tune-custom-gym-env-config-params-with-pbt/10673/1 "2023-05-16T07:53:11Z")

</div>

I am training a RL model and trying to tune hyperparameters using PBT. by following this example:

[https://docs.ray.io/en/latest/tune/examples/pbt\_ppo\_example.html](https://docs.ray.io/en/latest/tune/examples/pbt_ppo_example.html)

It all works like a charm, but I can’t figure out how to tune my custom environment hyperparameters.  
PBT scheduler **hyperparam\_mutations** seems to accept only the algo hyperparameters (PPO in my case) but not the custom environment ones (see below, “base\_line\_degrees” is not sampled from range (0.1 , 2.0) , instead it always uses the default value (-1) set on the custom gym env init method :

```auto
register_env("custom_env", lambda config: custom_env(config))

env_config = {
    "base_line_degrees": 1
}

ppo_config = (
    PPOConfig()
    .environment(env="custom_env", env_config=env_config)
)

hyperparam_mutations = {
    "lambda" : lambda: random.uniform(0.9,1.0),
    "clip_param" : lambda: random.uniform(0.01, 0.5),
    "lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
    "base_line_degrees": lambda: random.uniform(0.1,2.0)
}

pbt = PopulationBasedTraining(
    time_attr="time_total_s",
    perturbation_interval=120,
    resample_probability=0.25,
    hyperparam_mutations=hyperparam_mutations
)

analysis = tune.run(
    "PPO", 
    metric = "episode_reward_mean", 
    mode = "max",
    scheduler = pbt,
    num_samples=1,
    config = ppo_config.to_dict()
)

```

I’m clearly not doing it right. It doesn’t seems to me that I’m passing PBT any way to access the env\_conf object, but I can’t see any example/documentation on how to do it.

Can custom gym hyperparams be trained with PBT? and if so, how should it be done?

Thank for your help : )

---

<div class="post-metadata">

**Author:** ![ihopethiswillfi](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@ihopethiswillfi](https://discuss.ray.io/u/ihopethiswillfi)\
**Post date:** [August 29, 2023, 7:06pm UTC](https://discuss.ray.io/t/tune-custom-gym-env-config-params-with-pbt/10673/2 "2023-08-29T19:06:48Z")

</div>

@PREJAN did you ever find the solution? I’m facing the same obstacle.

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [August 30, 2023, 9:55am UTC](https://discuss.ray.io/t/tune-custom-gym-env-config-params-with-pbt/10673/3 "2023-08-30T09:55:38Z")

</div>

Hi @PREJAN and @ihopethiswillfi,

apologies for the late reply here.

You can specify nested dicts in the hyperparameter mutations, like this:

```auto
pbt = PopulationBasedTraining(
    ...
    hyperparam_mutations={
        "lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
        "env_config": {
            "my_param": lambda: random.uniform(1.10, 1.50)
        }
    },
)

```

Just for reference, here’s a full running toy example with a custom cartpole env:

```auto

import random

from ray import train, tune
from ray.rllib.algorithms.ppo import PPO
from ray.tune.schedulers import PopulationBasedTraining

from gymnasium.envs.classic_control.cartpole import CartPoleEnv

class MyCartPoleEnv(CartPoleEnv):
    def __init__ (self, my_param):
        print("MY PARAM", my_param)
        super(). __init__ ()

pbt = PopulationBasedTraining(
    time_attr="time_total_s",
    perturbation_interval=4,
    resample_probability=1,
    # Specifies the mutations of these hyperparams
    hyperparam_mutations={
        "lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
        "env_config": {
            "my_param": lambda: random.uniform(1.10, 1.50)
        }
    },
)

tune.register_env("MyCartPole", MyCartPoleEnv)

tuner = tune.Tuner(
    PPO,
    run_config=train.RunConfig(
        name="pbt_cartpole",
    ),
    tune_config=tune.TuneConfig(
        scheduler=pbt,
        num_samples=8,
        metric="episode_reward_mean",
        mode="max",
    ),
    param_space={
        "env": "MyCartPole",
        "lr": 0.01,
        "env_config": {
            "my_param": 1.3
        }
    },
)
results = tuner.fit()

print("best hyperparameters: ", results.get_best_result().config)

```