Searching Across Environment Configurations

RLlib
1
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

What is the best way to search different environment configurations using Tune?
I found the env_config key, val that can be passed in with the config option of tune.run(), where a tune.<search> algorithm can be used to search across various settings. However, I’m training a multi-agent environment where the policies need to have the observation and action space given to it before tune.run() is executed. Currently I’m achieving this by creating a sample environment just before tune.run() and extracting the sample_env.observation_space and sample_env.action_space.
Unless some input validation or something is happening, I’m assuming the policies only need to know about the shape of the observations/actions. However, if the shape changes with each environment configuration, there is a mismatch between policy and environment and an error is thrown.

What is the proper way to handle this situation?

2

Hey @Shaun_Fattig! Just to clarify are you using RLlib here?

3

@amogkam Correct, using RLlib

4

Got it thanks. cc @gjoliver could take a look?

5

sorry I don’t get why you need to create your env before Tune runs.
can you register an env creation function for your env that takes in env_cfg and creates an corresponding env?
E.g.,

6

@gjoliver

I’m passing the policies into the ‘multiagent’ key of the Tune config parameter, and each policy requires observations and actions to be specified beforehand.

ma_config = {
    ...,
    policies = {
        'policy1': (policy_cls1, obs_space1, act_space1, config1),
        'policy2': (policy_cls2, obs_space2, act_space2, config2)
    }
}

tune.run(
    ...,
    config = {
        ...,
        'multiagent': ma_config
    }
)

I see in that example environment that PolicySpec is being used to initialize the policies. I’m not familiar with this class, so that may hold keys to my issues.

By the way, I’m currently using Ray 1.6

7

Any reason you are using a 6 month old version?

I think you can just specify None for the spaces, which RLlib would simply use the Env to determine space types:

Hope this helps.