Environments with VectorEnv not able to run in parallel

RLlib
1

I have created an custom environment class that uses the VectorEnv as a subclass using Cartpole-v1 base env as shown below,

class MockVectorEnv(VectorEnv):


    def __init__(self, num_envs):
        self.env = gym.make("CartPole-v1")
        super().__init__(
            observation_space=self.env.observation_space,
            action_space=self.env.action_space,
            num_envs=num_envs,
        )

    def vector_reset(self) -> List[EnvObsType]:
        obs = self.env.reset()
        return [obs for _ in range(self.num_envs)]

    def reset_at(self, index: Optional[int] = None) -> EnvObsType:
        return self.env.reset()

    def vector_step(self, actions) -> Tuple[List[EnvObsType], List[float], List[bool], List[EnvInfoDict]]:
        obs_batch, rew_batch, done_batch, info_batch = [], [], [], []
        for i in range(self.num_envs):
            obs, rew, done, info = self.env.step(actions[i])
            obs_batch.append(obs)
            rew_batch.append(rew)
            done_batch.append(done)
            info_batch.append(info)
        return obs_batch, rew_batch, done_batch, info_batch

    def get_sub_environments(self) -> List[EnvType]:
        return [self.env for _ in range(self.num_envs)]

But when I am trying to use this env and setting num_envs_per_worker:5 and remote_worker_envs:True , envs are not running in parallel as shown below. When I looked into the dashboard only one worker i.e only one cpu is being used.

if __name__ == "__main__":

    ray.init( ) #dashboard_host="0.0.0.0", dashboard_port=8265)
    tune.register_env("custom_vec_env", lambda env_ctx: MockVectorEnv(1))
    analysis = tune.run(
        "PPO",
        local_dir="./results/tb_logs_1/",
        stop={"timesteps_total": 129000},
        metric="episode_reward_mean",
        mode="max",
        config={
            "env": 'custom_vec_env',
            "use_gae": True,
            "num_workers": 1,  
            "num_envs_per_worker": 5,
            "rollout_fragment_length": 100,
            "sgd_minibatch_size": 128,
            "train_batch_size": 1000,
            "lr": 0.001,
            "gamma": 0.95,
            "entropy_coeff": 0.02,
            "num_sgd_iter": 10,
            "remote_worker_envs": True,
        },
    )

This was not the case when I used CartPole-v1 env directly without custom VectorEnv. Why am I not able to run the envs in parallel using custom VectorEnv? am I doing anything wrong? Please help me with this.

2

why are you setting MockVectorEnv(1) then “num_envs_per_worker: 5”?
I mean, doesn’t seem like you need a vector env yourself? you can just use CartPole and set num_envs_per_worker: 5, and RLlib will vectorize things for you.

3

Thank you for the reply, I have an custom env and for vectorization I used VectorEnv subclass, so before directly working with my env I am trying to understand it with the basic gym env CartPole-v1.
My understanding was that setting “num_envs_per_worker:5” means it uses the “MockVectorEnv(1)” and creates 5 sub environments and runs them in parallel. Am I understanding it in wrong way? should we use either “VectorEnv” or “num_envs_per_worker” not both at a time? Can not we run the environments in parallel using VectorEnv subclass.? Plese help me on getting clarity on this.

4

ah, no problem.
so normally users don’t have to deal with vectorization themselves.
they just need to provide a single gym env, and when we convert your env into a RLlib BaseEnv, we will create multiple copies, and vectorize it for you.
for example:

self.num_envs is basically controlled by num_envs_per_worker parameter.

5

Thank you for the clarification. Then is it not possible to run environments in parallel with our custom env using VectorEnv? When I am setting “MockVectorEnv(5)” I can observe that training is very fast but I am not able to see multiple cores running in parallel. Am I missing any logic to understand or am I implementing my “MockVectorEnv” in wrong way. Please help me

6

RLlib will wrap your environment automatically, if you tell it that num_envs_per_worker should be > 1. Is training really “much” faster then if you would let RLLib serialize your env?

7

And fast in the sense of reaching higher rewards in less wall clock time? Or per sample?

8

MockVectorEnv is more for demonstration purpose. Probably should not use it in your work.
The multiple vectorized envs on a single worker are stepped through sequentially, so that’s why you notice only 1 CPU is utilized.
Parallelism comes in 2 ways:

  1. if you set num_envs_per_worker=1, but num_workers=5, then these multiple workers will sample their envs in parallel. This is the most common way to scale up your simulation.
  2. if you set num_envs_per_worker>1, then remote_worker_envs=True, that will make each env a remote actor, and will also allow you to run them in parallel.
    But I think you should try the num_workers route first.
    hope this helps.
1 Like
9

Thank you for the reply, I kept stop criteria as timesteps_total, I was just looking into the time than reward , I know it is important to look at reward and all the parameters are tuned to maximize the reward. I request you to take a look in the below table which makes you to clearly understand what I mean

S.no MockVectorEnv() num_workers num_envs_per_worker remote_wroker_envs timesteps_total total time(s) reward episode_reward_max episode_reward_min episode_len_mean
1 1 1 1 — 100000 128.558 392.38 500 12 392.38
2 1 1 3 FALSE 100000 120.078 342.68 500 29 342.68
3 1 1 3 TRUE 100000 118.02 408.52 500 46 408.56
4 3 1 3 TRUE 100000 57.99 98.14 500 5 98.63
5 3 1 3 FALSE 100000 58.50 193.47 500 11 193.75

considering all the above cases , in ray dashboard I am able to see that out of 12 cpu core only 2 of them are used justifying num_workers+1. Usually when “remote_worker_envs:True” then total_num_cpus= (num_worker+1)+(num_envs_per_worker) in this case each env is taking saperate cpu core to run in parallel and there will be a difference in the total time(s) too between running environments parallel and sequential. In my case I am not able to find these differences. If I am wrong please correct me and help me in understanding it.

10

Thank you for such good explanation. Can I know what exactly is the differenece between the num_worker and ray actor?

11

RLlib workers are implemented using Ray actors.