By default, does DQN support discrete observation spaces?

RLlib
1

Sorry if this is a beginner question.

Running DQN on FrozenLake-v1 results in the following error: ValueError: No default encoder config for obs space=Discrete(16), lstm=False found.), taking actor 1 out of service.

The DQN rain rainbow catalog source code also doesnt seem to define an encoder for discrete observation spaces: ray/rllib/algorithms/dqn/dqn_rainbow_catalog.py at releases/2.40.0 · ray-project/ray · GitHub

This seems quite surprising to me. Could it be an error related to the catalog and the new api stack?

Any advice would be appreciated. thanks!

2

Hi @itwasabhi and welcome to the forum. Your guess is correct. We do not provide default encoders for discrete observation spaces.

The usual workaround is to either

  • define a custom RLModule (see here for an example) or
  • or use the FlattenObservations connector to convert discrete observations to one-hot encoded observations (see below for the example).
from ray.rllib.algorithms.dqn import DQNConfig
from ray.rllib.connectors.env_to_module import FlattenObservations

config = (
    DQNConfig()
    ...
    .env_runners(
          env_to_module_connector=lambda env: FlattenObservations(),
    )
)

As an explanation:

The FlattenObservations connector is an EnvToModule connector and is run in the corresponding EnvToModulePipeline when sending the observations from the environment to the RLModule (that holds the policy). Any required transformation of data from the environment can be included into these EnvToModule connectors.

Note, the FlattenObservations rewrites observations from the enviornment - here discrete observations into one-hot encoded ones - back into the episodes, i.e. we also train on these one-hot encoded observations.

3

I see, thanks for the info!