Issue Reproducing results

narik11 · June 14, 2021, 12:00am

Hi, Am having issue to reproduce the exact results. The seed for the environment is set properly based on Reproducible training - setting seeds for all workers / environments

and also shuffle_sequences has been set to ‘False’ (just
in case). is there anything am missing ??

Lauritowal · June 14, 2021, 9:19am

I can be wrong here, but I don’t think it is necessary to set shuffle_sequences at all. It was not necessary for me

Did you set also the seed for the action_space in your environment as described in the end of the discussion here : Reproducible training - setting seeds for all workers / environments - #15 by Lauritowal

env.action_space.seed(RANDOM_SEED)

mannyv · June 14, 2021, 11:03am

I am guessing you are referring to the first empty list in the result. All the others look to be the same. This is expected behavior.

The reason is because the default config has a key called “create_env_on_driver” that is False by default. The behavior of foreach_env checks the driver and if it does not have an env it returns an empty list.

In the code snippet below local_worker/local_results refers to the driver.

github.com

ray-project/ray/blob/d89fb82bfb93b7a069b74d46fcacf462302881b1/rllib/evaluation/worker_set.py#L233-L251

    
      
          def foreach_env(self, func: Callable[[BaseEnv], List[T]]) -> List[List[T]]:
              """Apply `func` to all workers' (unwrapped) environments.
          
          
    `func` takes a single unwrapped env as arg.
          
          
    Args:
                  func (Callable[[BaseEnv], T]): A function - taking a BaseEnv
                      object as arg and returning a list of return values over envs
                      of the worker.
          
          
    Returns:
                  List[List[T]]: The list (workers) of lists (environments) of
                      results.
              """
              local_results = [self.local_worker().foreach_env(func)]
              ray_gets = []
              for worker in self.remote_workers():
                  ray_gets.append(worker.foreach_env.remote(func))
              return local_results + ray.get(ray_gets)

narik11 · June 14, 2021, 1:16pm

just want to make sure if shuffle_sequences has an impact.
I haven’t set this “env.action_space.seed(RANDOM_SEED)” in the environment. I did this in the past while using SB3 (when we instantiate an object of the class and that too when we use random actions), can try here as well and confirm…

narik11 · June 14, 2021, 1:23pm

thanks for looking into it… attached the screen shot to make sure the seed value is the same in all the environments.

Topic		Replies	Views
Seed of envs while using multi works/vector envs RLlib	2	310	October 30, 2021
Reproducible training - setting seeds for all workers / environments RLlib	20	5897	May 24, 2023
How I can generate the exactly same results in the rllib? RLlib	1	465	November 23, 2021
Evolution strategies - make reproducible RLlib	1	515	July 14, 2021
How do I set seed (randomize) for each rollout (for a given environment, worker and vector environment)? RLlib	0	268	August 28, 2023

Issue Reproducing results

Related topics