Shared backbone layers and unshared layers in a model

RLlib
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I got a multiple env like this.

 envs = ["env1","env2","env3", "env4","env5","env6","env7","env8","env9"]

    class MultiTaskEnv(gym.Env):
        def __init__(self, env_config):
            for i in range(len(envs)):
                if env_config.worker_index%9==i:
                    self.env = gym.make(envs[i], full_action_space=True)
                    self.name= envs[i]

For example, there are 2 parts in the default model (visionnet), cnn layer and fc layer. When I train the agent on a multienv setup, both cnn layer and fc layer weights are shared across all the different environments. But now I want to have the visionnet model as a backbone that is shared across all the envs, and additional fc layers for each of the different envs (that are not shared across the envs)

Thanks!

2

Interesting question-I think the easiest way for you to do this is to (1) concatenate the observations from all the environments inside your env.step method and (2) Create a custom model similar to ray/visionnet.py at master · ray-project/ray · GitHub that takes in all 9 observations, passes them sequentially (or as a batch) through a single CNN layer, then through each of the individiual fc layers per env, and concatenate the results back together.

If you can provide a repro script and more information about your setup, we may be able to provide better feedback.