Assert agent_key not in self.agent_collectors

I’m using ray/rllib 1.2.0 and I’m hitting this error after 100 training iterations.

 Failure # 1 (occurred at 2021-03-31_11-10-22)
Traceback (most recent call last):
  File "/opt/miniconda/lib/python3.7/site-packages/ray/tune/", line 586, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/opt/miniconda/lib/python3.7/site-packages/ray/tune/", line 609, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/opt/miniconda/lib/python3.7/site-packages/ray/_private/", line 47, in wrapper
    return func(*args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/ray/", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AssertionError): e[36mray::PPO.train_buffered()e[39m (pid=264, ip=
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/opt/miniconda/lib/python3.7/site-packages/ray/tune/", line 167, in train_buffered
    result = self.train()
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/agents/", line 526, in train
    raise e
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/agents/", line 515, in train
    result = Trainable.train(self)
  File "/opt/miniconda/lib/python3.7/site-packages/ray/tune/", line 226, in train
    result = self.step()
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/agents/", line 157, in step
    evaluation_metrics = self._evaluate()
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/agents/", line 778, in _evaluate
    for w in self.evaluation_workers.remote_workers()
  File "/opt/miniconda/lib/python3.7/site-packages/ray/_private/", line 47, in wrapper
    return func(*args, **kwargs)
ray.exceptions.RayTaskError(AssertionError): e[36mray::RolloutWorker.sample()e[39m (pid=375, ip=
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/evaluation/", line 662, in sample
    batches = []
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/evaluation/", line 95, in next
    batches = [self.get_data()]
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/evaluation/", line 224, in get_data
    item = next(self.rollout_provider)
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/evaluation/", line 620, in _env_runner
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/evaluation/", line 1198, in _process_observations_w_trajectory_view_api
    new_episode.length - 1, filtered_obs)
  File "/opt/miniconda/lib/python3.7/site-packages/ray/rllib/evaluation/collectors/", line 487, in add_init_obs
    assert agent_key not in self.agent_collectors

There is nothing very particular going on in my code, except few customizations (model, env, action distribution). I’ve checked memory usage and it is mostly below 20%. Does anyone know what it can be? Shall I open a github ticket?

Hey @GattiPinheiro . Could you try this on the latest master? This was fixed and merged yesterday.
PR: [RLlib] Issue: Agent_id -> Policy_id mapping should not need to be fixed between episodes. by sven1977 · Pull Request #15020 · ray-project/ray · GitHub


thanks for the feedback! I’m trying to install the nightly run following instructions from Installation, i.e.

$ python -m pip install

but I can’t make it work. The job is in the PENDING state forever without any errors. The best feedback I have is the following warning

/opt/miniconda/lib/python3.7/site-packages/ray/autoscaler/_private/ FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via pip install ‘ray[full]’. Please update your install command. "update your install command.", FutureWarning)

What am I missing?

I think this is a warning you can ignore. If the dashboard is not installed, you can always (re)start ray on your server via:
ray start --head --include-dashboard false

Hello @sven1977, I finally had some time to do the testing. As suggested, I installed ray from the today’s nightly build, but the error is still being raised after a couple of training iterations. I will try to downgrade to version 1.1.0 to check if it helps.

I created the issue at [rllib] Assert agent_key not in self.agent_collectors · Issue #15297 · ray-project/ray · GitHub. Good news is that I can reproduce the issue using only standard optimization algorithms and environment implementation.