Custom Multi ENV with DQN: AttributeError: 'NoneType' object has no attribute 'build_encoder'

RLlib
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I have a custom Multi Agent Environment inheriting from MultiAgentEnv. Simply, it has two agents who must guess a number from 1 to 100. The environment also picks a number from 1 to 100 (always 42). If the agent gets it correct, reward +1, 0 otherwise.

When building my algorithm with DQN (or really any other algorithm), I get the following error:

File “/rayTest/.venv/lib/python3.12/site-packages/ray/rllib/algorithms/dqn/default_dqn_rl_module.py”, line 52, in setup
self.encoder = self.catalog.build_encoder(framework=self.framework)
^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: ‘NoneType’ object has no attribute ‘build_encoder’

To me it appears to the catalog (?) not being defined. I have tried forcing this explicitly to no avail :cry:

My config is as follows:

from ray.rllib.algorithms.dqn import DQNConfig
from ray import train
from ray.rllib.algorithms.dqn.dqn_catalog import DQNCatalog

config = (
    DQNConfig()
    .environment(env=HI_Environment, env_config=env_config)
    .multi_agent(
        policies={"policy_alice", "policy_bob"},
        policy_mapping_fn=lambda agent_id: f"policy_{agent_id}",
        policies_to_train=["policy_alice", "policy_bob"],
    )
    .rl_module(
        rl_module_spec=MultiRLModuleSpec(rl_module_specs={
            "policy_alice": RLModuleSpec(observation_space=Discrete(100), action_space=Discrete(100)),
            "policy_bob": RLModuleSpec(observation_space=Discrete(100), action_space=Discrete(100)),
            
        }),
    )
    .api_stack(
        enable_rl_module_and_learner=True,
        enable_env_runner_and_connector_v2=True,
    )
    .framework("torch")
    .env_runners(num_env_runners=1)
    .training(dueling=False, double_q=False)
)

algo = config.build()

Running Python 3.12.8 with
pygame == 2.6.1
pettingzoo == 1.24.3
tqdm == 4.67.1 # progress bar
ray[train,tune,rllib,default] == 2.41.0
pillow == 11.1.0
torch == 2.5.1
ipykernel == 6.29.5
ipywidgets == 8.1.5

Appreciate any guidance on what is likely something obvious except to me!

thanks

2

Hi @Advertise-Here , thanks for raising this issue and welcome to the forum.

When giving a custom module you need to provide a ray.rllib.core.models.catalog.Catalog The definition of a custom RLModule is here, however, not required - as long as you do not want to use a custom module you programmed.

So, if you could try again without using the .rl_module method in the configuration, this should work:

config = (
    DQNConfig()
    .environment(env=HI_Environment, env_config=env_config)
    .multi_agent(
        policies={"policy_alice", "policy_bob"},
        policy_mapping_fn=lambda agent_id: f"policy_{agent_id}",
        policies_to_train=["policy_alice", "policy_bob"],
    )
    .api_stack(
        enable_rl_module_and_learner=True,
        enable_env_runner_and_connector_v2=True,
    )
    .framework("torch")
    .env_runners(num_env_runners=1)
    .training(dueling=False, double_q=False)
)