# KeyError: 'obs' In tower 0 on device cpu **URL:** \ **Category:** Configure Algorithm, Training, Evaluation, Scaling\ **Created:** [May 8, 2024, 2:41pm UTC](https://discuss.ray.io/t/keyerror-obs-in-tower-0-on-device-cpu/14616 "2024-05-08T14:41:41Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Federica\_Tonti](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/federica_tonti/32/6094_2.png) [@Federica\_Tonti](https://discuss.ray.io/u/Federica_Tonti)\ **Post date:** [May 8, 2024, 2:41pm UTC](https://discuss.ray.io/t/keyerror-obs-in-tower-0-on-device-cpu/14616/1 "2024-05-08T14:41:41Z")
Traceback (most recent call last): File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/air/execution/\_internal/event\_manager.py”, line 110, in resolve\_future result = ray.get(future) ^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/\_private/auto\_init\_hook.py”, line 21, in auto\_init\_wrapper return fn(\*args, \*\*kwargs) ^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/\_private/client\_mode\_hook.py”, line 103, in wrapper return func(\*args, \*\*kwargs) ^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/\_private/worker.py”, line 2623, in get values, debugger\_breakpoint = worker.get\_objects(object\_refs, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/\_private/worker.py”, line 861, in get\_objects raise value.as\_instanceof\_cause() ray.exceptions.RayTaskError(ValueError): ray::PPO.train() (pid=285359, ip=130.237.233.123, actor\_id=1a4964d041decbd253209b4601000000, repr=PPO) File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/ppo/ppo\_torch\_policy.py”, line 84, in loss logits, state = model(train\_batch) ^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/models/modelv2.py”, line 244, in **call** input\_dict[“obs”], self.obs\_space, self.framework ~~~~~~~~~~ ^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/sample\_batch.py”, line 950, in **getitem** value = dict. **getitem** (self, key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyError: ‘obs’ The above exception was the direct cause of the following exception: ray::PPO.train() (pid=285359, ip=130.237.233.123, actor\_id=1a4964d041decbd253209b4601000000, repr=PPO) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/tune/trainable/trainable.py”, line 331, in train raise skipped from exception\_cause(skipped) File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/tune/trainable/trainable.py”, line 328, in train result = self.step() ^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py”, line 878, in step train\_results, train\_iter\_ctx = self.\_run\_one\_training\_iteration() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py”, line 3156, in \_run\_one\_training\_iteration results = self.training\_step() ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/ppo/ppo.py”, line 426, in training\_step return self.\_training\_step\_old\_and\_hybrid\_api\_stacks() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/ppo/ppo.py”, line 601, in \_training\_step\_old\_and\_hybrid\_api\_stacks train\_results = multi\_gpu\_train\_one\_step(self, train\_batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/execution/train\_ops.py”, line 176, in multi\_gpu\_train\_one\_step results = policy.learn\_on\_loaded\_batch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/torch\_policy\_v2.py”, line 831, in learn\_on\_loaded\_batch return self.learn\_on\_batch(batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/utils/threading.py”, line 24, in wrapper return func(self, \*a, \*\*k) ^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/torch\_policy\_v2.py”, line 712, in learn\_on\_batch grads, fetches = self.compute\_gradients(postprocessed\_batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/utils/threading.py”, line 24, in wrapper return func(self, \*a, \*\*k) ^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/torch\_policy\_v2.py”, line 924, in compute\_gradients tower\_outputs = self.\_multi\_gpu\_parallel\_grad\_calc([postprocessed\_batch]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/torch\_policy\_v2.py”, line 1421, in \_multi\_gpu\_parallel\_grad\_calc raise last\_result[0] from last\_result[1] ValueError: obs tracebackTraceback (most recent call last): File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/torch\_policy\_v2.py”, line 1336, in \_worker self.loss(model, self.dist\_class, sample\_batch) File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/ppo/ppo\_torch\_policy.py”, line 84, in loss logits, state = model(train\_batch) ^^^^^^^^^^^^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/models/modelv2.py”, line 244, in **call** input\_dict[“obs”], self.obs\_space, self.framework ~~~~~~~~~~ ^^^^^^^ File “/scratch/ftonti/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/sample\_batch.py”, line 950, in **getitem** value = dict. **getitem** (self, key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyError: ‘obs’ In tower 0 on device cpu I get the following error when running the script on Linux Debian, but it runs on my Mac M2. The environment is the same, the script is exactly the same. ☹ I tried milion times but I could not solve the issue. Can you give me some help? Thanks a lot! ---
**Author:** ![Vasiliy\_Shimko](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vasiliy_shimko/32/6115_2.png) [@Vasiliy\_Shimko](https://discuss.ray.io/u/Vasiliy_Shimko)\ **Post date:** [May 11, 2024, 2:18pm UTC](https://discuss.ray.io/t/keyerror-obs-in-tower-0-on-device-cpu/14616/2 "2024-05-11T14:18:37Z")
I have a similar problem. I have an environment that I’ve created and successfully trained with rllib 2.10.0 on ArchLinux, but after upgrading to rllib 2.21 it stopped working. Tried downgrading to at least version 2.11 and still had the issue. Downgrading back to 2.10 solved the issue. I have another environment that I’ve created and trained with version 2.10 and had no problems with on 2.21. I think it may have something to do with the first model (the one that won’t work) having a Dict within Dict in its observation space. ---
**Author:** ![rgbarishian](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rgbarishian/32/6628_2.png) [@rgbarishian](https://discuss.ray.io/u/rgbarishian)\ **Post date:** [October 10, 2024, 3:26am UTC](https://discuss.ray.io/t/keyerror-obs-in-tower-0-on-device-cpu/14616/3 "2024-10-10T03:26:52Z")
In Ray 2.30, this can be patched by either reducing `rollout_fragment_length` or increasing `sample_timeout_s` within `env_runners`. This error is caused by a timeout in data collection of rollout fragments but these two modifications can mitigate it. Ensure `rollout_fragment_length` is evenly divisible by `train_batch_size` within 10%. You’ll see a different error if this isn’t accurate.