# \[Rllib\]\[Bug\] Custom Multi-agent Environment Observation Space " does not contain returned observation after a reset"

**URL:** <https://discuss.ray.io/t/rllib-bug-custom-multi-agent-environment-observation-space-does-not-contain-returned-observation-after-a-reset/5665>\
**Category:** RLlib\
**Created:** [April 4, 2022, 8:28am UTC](https://discuss.ray.io/t/rllib-bug-custom-multi-agent-environment-observation-space-does-not-contain-returned-observation-after-a-reset/5665 "2022-04-04T08:28:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ZKBig](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zkbig/32/2285_2.png) [@ZKBig](https://discuss.ray.io/u/ZKBig)\
**Post date:** [April 4, 2022, 8:28am UTC](https://discuss.ray.io/t/rllib-bug-custom-multi-agent-environment-observation-space-does-not-contain-returned-observation-after-a-reset/5665/1 "2022-04-04T08:28:44Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

> <https://github.com/ray-project/ray/issues/23368>
>
> \### Search before asking
> 
> \- \[X\] I searched the \[issues\](https://github.com/ray-p…roject/ray/issues) and found no similar issues.
> 
> 
> \### Ray Component
> 
> RLlib
> 
> \### Issue Severity
> 
> High: It blocks me to complete my task.
> 
> \### What happened + What you expected to happen
> 
> In my project, I am supposed to first train a single automatic vehicle to drive in a custom traffic network, and then applying this trained policy for multiple automatic vehicles. According to the documentation of rllib multi-agent, I need to modify the environment for single agent to an updated environment for multi-agent to achieve this function. In the meantime, each vehicle should be assigned with an isolated policy. 
> 
> Because my environment for single vehicle is based on the gym class, then according to the documentation of rllib, the following contents are changed for obtaining a multi-agent environment:
> \*\*1. For the 'step()' function:\*\*
> The returned observation is modified to contain all the rl vehicles, and each rl vehicle is assigned with a particular observation value with the help of 'dict' in python.
> Note that the returned values of reward and done should also be extended to meet multi-agent demand in virtu of 'dict' in python.
> 
> \*\*2. The input attribute 'actions' of 'step()' function:\*\*
> After modifying, the input actions contains all the actions for the rl vehicles, then assign the action to each rl vehicle according to the name of rl vehicles in 'actions'.
> 
> \*\*3. The modified multi-agent environment class inherits the 'MultiAgentEnv' in rllib as well.\*\*
> 
> It is important to point out that the observation space and the action space is not changed, which remains the same as the single vehicle following the documents. 
> 
> The above is all the changes of my custom environment class. When making the preparation for training the multi-agent, I added the configuration for multi-agent as well, which is shown as follows:
> \<img width="579" alt="code1" src="https://user-images.githubusercontent.com/71830568/159216912-39f016f0-0c99-4bd4-9af1-3b3654b26298.png"\>
> 
> Then, I use tune.run(...) to train my own model; however, an unexpected error occurred, which is shown as follows:
> \`Failure # 1 (occurred at 2022-03-16\_15-18-49)
> Traceback (most recent call last):
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/trial\_runner.py", line 924, in \_process\_trial
> results = self.trial\_executor.fetch\_result(trial)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/ray\_trial\_executor.py", line 787, in fetch\_result
> result = ray.get(trial\_future\[0\], timeout=DEFAULT\_GET\_TIMEOUT)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/\_private/client\_mode\_hook.py", line 105, in wrapper
> return func(\*args, \*\*kwargs)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/worker.py", line 1715, in get
> raise value
> ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, \[36mray::PPO.\_\_init\_\_()\[39m (pid=1511, ip=127.0.0.1)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py", line 102, in \_\_init\_\_
> Trainer.\_\_init\_\_(self, config, env, logger\_creator,
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 661, in \_\_init\_\_
> super().\_\_init\_\_(config, logger\_creator, remote\_checkpoint\_dir,
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/trainable.py", line 121, in \_\_init\_\_
> self.setup(copy.deepcopy(self.config))
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py", line 113, in setup
> super().setup(config)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 764, in setup
> self.\_init(self.config, self.env\_creator)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/agents/trainer\_template.py", line 136, in \_init
> self.workers = self.\_make\_workers(
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1727, in \_make\_workers
> return WorkerSet(
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker\_set.py", line 87, in \_\_init\_\_
> remote\_spaces = ray.get(self.remote\_workers(
> ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, \[36mray::RolloutWorker.\_\_init\_\_()\[39m (pid=1513, ip=127.0.0.1)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout\_worker.py", line 463, in \_\_init\_\_
> \_validate\_env(self.env, env\_context=self.env\_context)
> File "/Users/XXXXX/opt/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout\_worker.py", line 1700, in \_validate\_env
> raise EnvError(
> ray.rllib.utils.error.EnvError: Env's \`observation\_space\` Dict(action\_mask:Box(\[0. 0. 0. 0.\], \[1. 1. 1. 1.\], (4,), float32), position:Box(\[\[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]
> \[-1.\]\], \[\[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]
> \[1.\]\], (38, 1), float32), real\_observation:Box(\[\[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]
> \[-1. -1. -1. -1. -1.\]\], \[\[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]
> \[1. 1. 1. 1. 1.\]\], (38, 5), float32)) does not contain returned observation after a reset ({'rl0': {'real\_observation': array(\[\[0. , 0. , 0.03572819, 0.34533966, 1. \],
> \[0. , 0. , 0.0067462 , 0.34533966, 0. \],
> \[0. , 0. , 0.04358931, 0.6188486 , 0. \],
> \[0. , 0. , 0.17568235, 0.6188486 , 0. \],
> \[0. , 0. , 0.05400503, 0.9641883 , 0. \],
> \[0. , 0. , 0.30195004, 0.9641883 , 0. \],
> \[0. , 0. , 0.3528907 , 0.63266224, 0. \],
> \[0. , 0. , 0.20100671, 0.63266224, 0. \],
> \[0. , 0. , 0.34633353, 0.9641883 , 0. \],
> \[0. , 0. , 0.19366628, 0.9641883 , 0. \],
> \[0. , 0. , 0.3083996 , 0.6188486 , 0. \],
> \[0. , 0. , 0.35600442, 0.6188486 , 0. \],
> \[0. , 0. , 0.3648858 , 0.6188486 , 0. \],
> \[0. , 0. , 0.4674277 , 0.6188486 , 0. \],
> \[0. , 0. , 0.46439698, 0.9780019 , 0. \],
> \[0. , 0. , 0.3666407 , 0.9780019 , 0. \],
> \[0. , 0. , 0.32016182, 0.9641883 , 0. \],
> \[0. , 0. , 0.57191485, 0.9641883 , 0. \],
> \[0. , 0. , 0.59129936, 0.9641883 , 0. \],
> \[0. , 0. , 0.3743161 , 0.9641883 , 0. \],
> \[0. , 0. , 0.5748492 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6044079 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6679318 , 0.9641883 , 0. \],
> \[0. , 0. , 0.48739564, 0.9641883 , 0. \],
> \[0. , 0. , 0.6068588 , 0.6188141 , 0. \],
> \[0. , 0. , 0.67606044, 0.6188141 , 0. \],
> \[0. , 0. , 0.5922788 , 1. , 0. \],
> \[0. , 0. , 0.8389112 , 1. , 0. \],
> \[0. , 0. , 0.8550952 , 0.98950166, 0. \],
> \[0. , 0. , 0.62028337, 0.98950166, 0. \],
> \[0. , 0. , 0.9088409 , 0.9887074 , 0. \],
> \[0. , 0. , 0.6943932 , 0.9887074 , 0. \],
> \[0. , 0. , 0.8472192 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8698804 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8703939 , 0.6165694 , 0. \],
> \[0. , 0. , 0.9204792 , 0.6165694 , 0. \],
> \[0. , 0. , 1. , 0.32057878, 0. \],
> \[0. , 0. , 0.93785423, 0.32057878, 0. \]\],
> dtype=float32), 'position': array(\[\[0.06163448\],
> \[0.06495931\],
> \[0.18669344\],
> \[0.1899465 \],
> \[0.29732072\],
> \[0.29967514\],
> \[0.35336545\],
> \[0.35480976\],
> \[0.386706 \],
> \[0.3927177 \],
> \[0.40525863\],
> \[0.4100809 \],
> \[0.4891037 \],
> \[0.4921863 \],
> \[0.5323883 \],
> \[0.53808206\],
> \[0.5624943 \],
> \[0.56471974\],
> \[0.5975516 \],
> \[0.60240877\],
> \[0.65928924\],
> \[0.664609 \],
> \[0.6909546 \],
> \[0.69610846\],
> \[0.7109397 \],
> \[0.7154712 \],
> \[0.8341697 \],
> \[0.8350319 \],
> \[0.8551079 \],
> \[0.85537493\],
> \[0.9175007 \],
> \[0.9190277 \],
> \[0.9268189 \],
> \[0.93248606\],
> \[0.96099406\],
> \[0.9667964 \],
> \[0.9978848 \],
> \[1. \]\], dtype=float32), 'action\_mask': \[1, 1, 0, 0\]}, 'rl1': {'real\_observation': array(\[\[0. , 0. , 0.03572819, 0.34533966, 1. \],
> \[0. , 0. , 0.0067462 , 0.34533966, 0. \],
> \[0. , 0. , 0.04358931, 0.6188486 , 0. \],
> \[0. , 0. , 0.17568235, 0.6188486 , 0. \],
> \[0. , 0. , 0.05400503, 0.9641883 , 0. \],
> \[0. , 0. , 0.30195004, 0.9641883 , 0. \],
> \[0. , 0. , 0.3528907 , 0.63266224, 0. \],
> \[0. , 0. , 0.20100671, 0.63266224, 0. \],
> \[0. , 0. , 0.34633353, 0.9641883 , 0. \],
> \[0. , 0. , 0.19366628, 0.9641883 , 0. \],
> \[0. , 0. , 0.3083996 , 0.6188486 , 0. \],
> \[0. , 0. , 0.35600442, 0.6188486 , 0. \],
> \[0. , 0. , 0.3648858 , 0.6188486 , 0. \],
> \[0. , 0. , 0.4674277 , 0.6188486 , 0. \],
> \[0. , 0. , 0.46439698, 0.9780019 , 0. \],
> \[0. , 0. , 0.3666407 , 0.9780019 , 0. \],
> \[0. , 0. , 0.32016182, 0.9641883 , 0. \],
> \[0. , 0. , 0.57191485, 0.9641883 , 0. \],
> \[0. , 0. , 0.59129936, 0.9641883 , 0. \],
> \[0. , 0. , 0.3743161 , 0.9641883 , 0. \],
> \[0. , 0. , 0.5748492 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6044079 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6679318 , 0.9641883 , 0. \],
> \[0. , 0. , 0.48739564, 0.9641883 , 0. \],
> \[0. , 0. , 0.6068588 , 0.6188141 , 0. \],
> \[0. , 0. , 0.67606044, 0.6188141 , 0. \],
> \[0. , 0. , 0.5922788 , 1. , 0. \],
> \[0. , 0. , 0.8389112 , 1. , 0. \],
> \[0. , 0. , 0.8550952 , 0.98950166, 0. \],
> \[0. , 0. , 0.62028337, 0.98950166, 0. \],
> \[0. , 0. , 0.9088409 , 0.9887074 , 0. \],
> \[0. , 0. , 0.6943932 , 0.9887074 , 0. \],
> \[0. , 0. , 0.8472192 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8698804 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8703939 , 0.6165694 , 0. \],
> \[0. , 0. , 0.9204792 , 0.6165694 , 0. \],
> \[0. , 0. , 1. , 0.32057878, 0. \],
> \[0. , 0. , 0.93785423, 0.32057878, 0. \]\],
> dtype=float32), 'position': array(\[\[0.06163448\],
> \[0.06495931\],
> \[0.18669344\],
> \[0.1899465 \],
> \[0.29732072\],
> \[0.29967514\],
> \[0.35336545\],
> \[0.35480976\],
> \[0.386706 \],
> \[0.3927177 \],
> \[0.40525863\],
> \[0.4100809 \],
> \[0.4891037 \],
> \[0.4921863 \],
> \[0.5323883 \],
> \[0.53808206\],
> \[0.5624943 \],
> \[0.56471974\],
> \[0.5975516 \],
> \[0.60240877\],
> \[0.65928924\],
> \[0.664609 \],
> \[0.6909546 \],
> \[0.69610846\],
> \[0.7109397 \],
> \[0.7154712 \],
> \[0.8341697 \],
> \[0.8350319 \],
> \[0.8551079 \],
> \[0.85537493\],
> \[0.9175007 \],
> \[0.9190277 \],
> \[0.9268189 \],
> \[0.93248606\],
> \[0.96099406\],
> \[0.9667964 \],
> \[0.9978848 \],
> \[1. \]\], dtype=float32), 'action\_mask': \[1, 1, 0, 0\]}, 'rl2': {'real\_observation': array(\[\[0. , 0. , 0.03572819, 0.34533966, 1. \],
> \[0. , 0. , 0.0067462 , 0.34533966, 0. \],
> \[0. , 0. , 0.04358931, 0.6188486 , 0. \],
> \[0. , 0. , 0.17568235, 0.6188486 , 0. \],
> \[0. , 0. , 0.05400503, 0.9641883 , 0. \],
> \[0. , 0. , 0.30195004, 0.9641883 , 0. \],
> \[0. , 0. , 0.3528907 , 0.63266224, 0. \],
> \[0. , 0. , 0.20100671, 0.63266224, 0. \],
> \[0. , 0. , 0.34633353, 0.9641883 , 0. \],
> \[0. , 0. , 0.19366628, 0.9641883 , 0. \],
> \[0. , 0. , 0.3083996 , 0.6188486 , 0. \],
> \[0. , 0. , 0.35600442, 0.6188486 , 0. \],
> \[0. , 0. , 0.3648858 , 0.6188486 , 0. \],
> \[0. , 0. , 0.4674277 , 0.6188486 , 0. \],
> \[0. , 0. , 0.46439698, 0.9780019 , 0. \],
> \[0. , 0. , 0.3666407 , 0.9780019 , 0. \],
> \[0. , 0. , 0.32016182, 0.9641883 , 0. \],
> \[0. , 0. , 0.57191485, 0.9641883 , 0. \],
> \[0. , 0. , 0.59129936, 0.9641883 , 0. \],
> \[0. , 0. , 0.3743161 , 0.9641883 , 0. \],
> \[0. , 0. , 0.5748492 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6044079 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6679318 , 0.9641883 , 0. \],
> \[0. , 0. , 0.48739564, 0.9641883 , 0. \],
> \[0. , 0. , 0.6068588 , 0.6188141 , 0. \],
> \[0. , 0. , 0.67606044, 0.6188141 , 0. \],
> \[0. , 0. , 0.5922788 , 1. , 0. \],
> \[0. , 0. , 0.8389112 , 1. , 0. \],
> \[0. , 0. , 0.8550952 , 0.98950166, 0. \],
> \[0. , 0. , 0.62028337, 0.98950166, 0. \],
> \[0. , 0. , 0.9088409 , 0.9887074 , 0. \],
> \[0. , 0. , 0.6943932 , 0.9887074 , 0. \],
> \[0. , 0. , 0.8472192 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8698804 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8703939 , 0.6165694 , 0. \],
> \[0. , 0. , 0.9204792 , 0.6165694 , 0. \],
> \[0. , 0. , 1. , 0.32057878, 0. \],
> \[0. , 0. , 0.93785423, 0.32057878, 0. \]\],
> dtype=float32), 'position': array(\[\[0.06163448\],
> \[0.06495931\],
> \[0.18669344\],
> \[0.1899465 \],
> \[0.29732072\],
> \[0.29967514\],
> \[0.35336545\],
> \[0.35480976\],
> \[0.386706 \],
> \[0.3927177 \],
> \[0.40525863\],
> \[0.4100809 \],
> \[0.4891037 \],
> \[0.4921863 \],
> \[0.5323883 \],
> \[0.53808206\],
> \[0.5624943 \],
> \[0.56471974\],
> \[0.5975516 \],
> \[0.60240877\],
> \[0.65928924\],
> \[0.664609 \],
> \[0.6909546 \],
> \[0.69610846\],
> \[0.7109397 \],
> \[0.7154712 \],
> \[0.8341697 \],
> \[0.8350319 \],
> \[0.8551079 \],
> \[0.85537493\],
> \[0.9175007 \],
> \[0.9190277 \],
> \[0.9268189 \],
> \[0.93248606\],
> \[0.96099406\],
> \[0.9667964 \],
> \[0.9978848 \],
> \[1. \]\], dtype=float32), 'action\_mask': \[1, 1, 0, 0\]}, 'rl3': {'real\_observation': array(\[\[0. , 0. , 0.03572819, 0.34533966, 1. \],
> \[0. , 0. , 0.0067462 , 0.34533966, 0. \],
> \[0. , 0. , 0.04358931, 0.6188486 , 0. \],
> \[0. , 0. , 0.17568235, 0.6188486 , 0. \],
> \[0. , 0. , 0.05400503, 0.9641883 , 0. \],
> \[0. , 0. , 0.30195004, 0.9641883 , 0. \],
> \[0. , 0. , 0.3528907 , 0.63266224, 0. \],
> \[0. , 0. , 0.20100671, 0.63266224, 0. \],
> \[0. , 0. , 0.34633353, 0.9641883 , 0. \],
> \[0. , 0. , 0.19366628, 0.9641883 , 0. \],
> \[0. , 0. , 0.3083996 , 0.6188486 , 0. \],
> \[0. , 0. , 0.35600442, 0.6188486 , 0. \],
> \[0. , 0. , 0.3648858 , 0.6188486 , 0. \],
> \[0. , 0. , 0.4674277 , 0.6188486 , 0. \],
> \[0. , 0. , 0.46439698, 0.9780019 , 0. \],
> \[0. , 0. , 0.3666407 , 0.9780019 , 0. \],
> \[0. , 0. , 0.32016182, 0.9641883 , 0. \],
> \[0. , 0. , 0.57191485, 0.9641883 , 0. \],
> \[0. , 0. , 0.59129936, 0.9641883 , 0. \],
> \[0. , 0. , 0.3743161 , 0.9641883 , 0. \],
> \[0. , 0. , 0.5748492 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6044079 , 0.6188486 , 0. \],
> \[0. , 0. , 0.6679318 , 0.9641883 , 0. \],
> \[0. , 0. , 0.48739564, 0.9641883 , 0. \],
> \[0. , 0. , 0.6068588 , 0.6188141 , 0. \],
> \[0. , 0. , 0.67606044, 0.6188141 , 0. \],
> \[0. , 0. , 0.5922788 , 1. , 0. \],
> \[0. , 0. , 0.8389112 , 1. , 0. \],
> \[0. , 0. , 0.8550952 , 0.98950166, 0. \],
> \[0. , 0. , 0.62028337, 0.98950166, 0. \],
> \[0. , 0. , 0.9088409 , 0.9887074 , 0. \],
> \[0. , 0. , 0.6943932 , 0.9887074 , 0. \],
> \[0. , 0. , 0.8472192 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8698804 , 0.6343544 , 0. \],
> \[0. , 0. , 0.8703939 , 0.6165694 , 0. \],
> \[0. , 0. , 0.9204792 , 0.6165694 , 0. \],
> \[0. , 0. , 1. , 0.32057878, 0. \],
> \[0. , 0. , 0.93785423, 0.32057878, 0. \]\],
> dtype=float32), 'position': array(\[\[0.06163448\],
> \[0.06495931\],
> \[0.18669344\],
> \[0.1899465 \],
> \[0.29732072\],
> \[0.29967514\],
> \[0.35336545\],
> \[0.35480976\],
> \[0.386706 \],
> \[0.3927177 \],
> \[0.40525863\],
> \[0.4100809 \],
> \[0.4891037 \],
> \[0.4921863 \],
> \[0.5323883 \],
> \[0.53808206\],
> \[0.5624943 \],
> \[0.56471974\],
> \[0.5975516 \],
> \[0.60240877\],
> \[0.65928924\],
> \[0.664609 \],
> \[0.6909546 \],
> \[0.69610846\],
> \[0.7109397 \],
> \[0.7154712 \],
> \[0.8341697 \],
> \[0.8350319 \],
> \[0.8551079 \],
> \[0.85537493\],
> \[0.9175007 \],
> \[0.9190277 \],
> \[0.9268189 \],
> \[0.93248606\],
> \[0.96099406\],
> \[0.9667964 \],
> \[0.9978848 \],
> \[1. \]\], dtype=float32), 'action\_mask': \[1, 1, 0, 0\]}})!
> 
> \`
> Note that the observation space defined in my environment is already a dictionary, which has 'action\_mask', 'real\_observation', 'position' three keys. However, this unexpected error happened when the environment was changed to multi-agent, and it seemed that it has the following two possible reasons:
> a) Some changes converted from single-agent to multi-agent of the environment are neglected;
> b) The corresponding multi-agent configuration has problem.
> However, I cannot figure out what wrong it is though I have tried my best.
> 
> The Expected Results:
> The model can be successfully trained in my multi-agent environment without any error.
> 
> 
> \### Versions / Dependencies
> 
> ray version: 1.11.0
> MacOS version: Monterey 12.2.1
> 
> \### Reproduction script
> 
> The codes are related to the external environment library; therefore, it is hard to attach the corresponding scripts. However, if it is necessary for solution, I will attach it as soon as possible. Thx!
> 
> \### Anything else
> 
> \_No response\_
> 
> \### Are you willing to submit a PR?
> 
> \- \[\] Yes I am willing to submit a PR!