2D Box Space flattening in ray 2.6.*

RLlib Configure Algorithm, Training, Evaluation, Scaling
1

Trying around with the latest action masking example by @arturn, I was trying to feed it into my problem. I noticed that in my gym envr, some static features (static = they are fed into neural network but do not change along episode) were not part of the “real_obs” or “observations”.

Now I get following error message by PPO Config in ray 2.6.1 (nightly build), tensorflow 2.11

ValueError: No default encoder config for obs space=Box(-1.0, 1.0, (1, 164), float64), lstm=False and attention=False found. 2D Box spaces are not supported. They should be either flattened to a 1D Box space or enhanced to be a 3D box space.

It seems that this is related to the following setting
.experimental(_disable_preprocessor_api=True)

What is the best way? Do a manual flattening with gym.spaces.flatten?

2

config.experimental(_disable_preprocessor_api=True) makes is so that you observations are not transformed. One standard transformation that RLlib does is to flatten nested observation spaces so that the model becomes less complex.

The observation space that you are looking at is not flattened. It is of size dimension (1, 164). Any chance you can just squeeze it?

1 Like
3

Useful hints, @arturn . During first checks it looks like that the FlattenObservation Wrapper delivered by gymnasium is quite handy. Or is there any hidden functionality of ray which prohibits usage of that wrapper?

4

Is it possible to have a mask for the observation_space as well? Imagine a situation where the observation_space is a space.Dict with keys that change throughout the steps. The action_space is also a space.Dict with the same keys as the observation_space .

1 Like
5

It seems that this thread has become stale somehow. Nevertheless, the need for advice or a bugfix solution is still there … push

6

When you open Ray Documentation, You will see the Ask AI box in the bottom right corner, try asking it, it knows all of the Ray Library. It is helping me alot. Can’t thank enough.

7

The solution I found consists of two parts:

  1. A gymnasium wrapper applied on top of the environment
  2. Usage of the classic env_creator to inject a wrapped environment to the algo config.

For Part 1 one possible implementation may look like that. Note that it is important to overwrite the observation_space, as this is checked by rlib.

class MyWrapper(TransformObservation):

    def __init__(self, env):
        super().__init__(env, self.__transform)

    def __transform(self, orig_obs):
        new_obs = orig_obs
        for b in new_obs.keys():
            if b not in ["instance_props"]:
                new_obs[b] = np.reshape(new_obs[b], -1)
        self.observation_space["real_obs"] = Box(0, 1, (len(new_obs["real_obs"]),))
        return new_obs

Part 2 adopts the gymnmasium-way of applying wrappers to envs:

def env_creator(env_config):
    env = MyEnv(env_config)
    env = MyWrapper(env)
    return env

register_env("env_wrapped", env_creator)