Preprocessor fails on observation vector

RLlib
1

I read RLlib Models, Preprocessors, and Action Distributions — Ray v2.0.0.dev0 and decided to write a wrapper around my environment, but it appears to cause issues with the preprocessor. I am using ray version (1.1.0). The output of my environment is as follows:

>>>[(k,v.shape) for k, v in observations]
[('objectgoal', (1,)), ('compass', (1,)), ('gps', (2,)), ('encoder', (512,))]
>>>print(self.observation_space.spaces)
Dict(compass:Box(-3.141592653589793, 3.141592653589793, (1,), float64), gps:Box(-3.4028234663852886e+38, 3.4028234663852886e+38, (2,), float32), objectgoal:Box(0, 20, (1,), int64), encoder:Box(-3.4028234663852886e+38, 3.4028234663852886e+38, (512,), float32))

However, I get the following error:

...
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/sampler.py", line 568, in _env_runner
    _process_observations_w_trajectory_view_api(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/sampler.py", line 1020, in _process_observations_w_trajectory_view_api
    prep_obs: EnvObsType = _get_or_raise(preprocessors,
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/preprocessors.py", line 240, in transform
    self.write(observation, array, 0)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/preprocessors.py", line 251, in write
    p.write(o, array, offset)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/preprocessors.py", line 174, in write
    array[offset:offset + self._size] = np.array(
ValueError: could not broadcast input array from shape (512) into shape (2)

Observations, their shapes, and matching preprocessors in rllib.models.preprocessors:

compass (1,) [<ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca88f7df70>, <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca89020070>, <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca890200d0>, <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca89020100>]
encoder (512,) [<ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca88f7df70>, <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca89020070>, <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca890200d0>, <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7fca89020100>]

EDIT: It appears the size of the preprocessor is incorrect:

>>>print(observation.shape, preprocessor.size)
(1,) 1
(512,) 2

EDIT2:
This appears to be a bug with the Dict preprocessors. If I remove other observations (compass, GPS, objectgoal) and encoder is the only observation, everything runs fine.

EDIT3:
I thought this might be due to the ordering of observation_space with respect to obs, so I sorted them equally yet the issue persists.

observation_space.spaces.keys(): odict_keys(['compass', 'gps', 'objectgoal', 'encoder'])
obs.keys(): dict_keys(['compass', 'gps', 'objectgoal', 'encoder'])

EDIT4:
Solved the issue by making obs a collections.OrderedDict instead of dict. Even though the dicts are sorted in my environment, and dicts are guaranteed to maintain insertion order in python3, they must be reconstructed in the ray worker in such a way that the order changes.

2 Likes
2

Great! Thanks for digging into this and posting the solution here @smorad !
Since py3.7, all (regular) dicts are always ordered by insertion(!) order. And yes, this may have caused problems when dicts were reconstructed using a different insertion order than the original. Using OrderedDict is a good solution for this problem.

1 Like
3

Hi @sven1977 just heads up that this is still a problem with ray 1.9.0 and Python 3.7.11.
@smorad’s solution still solves the problem though, thanks!

4

Hey @bmanczak , you could also disable preprocessing via the config._disable_preprocessor_api=True key. In this case, your model will receive the original observation from the env and may then do its own preprocessing of the different components.