ValueError: `RLModule(config=[RLModuleConfig])` has been deprecated

RLlib
1

Hello,

I encounter this error while passing the config to PPO:
(MultiAgentEnvRunner pid=28036) File “D:\Miniconda\envs\condaPars\Lib\site-packages\ray\rllib\core\rl_module\rl_module.py”, line 113, in build
(MultiAgentEnvRunner pid=28036) module = self.module_class(module_config)
(MultiAgentEnvRunner pid=28036) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(MultiAgentEnvRunner pid=28036) deprecation_warning(
(MultiAgentEnvRunner pid=28036) File “D:\Miniconda\envs\condaPars\Lib\site-packages\ray\rllib\utils\deprecation.py”, line 48, in deprecation_warning
(MultiAgentEnvRunner pid=28036) raise ValueError(msg)
(MultiAgentEnvRunner pid=28036) ValueError: RLModule(config=[RLModuleConfig]) has been deprecated. Use RLModule(observation_space=.., action_space=.., inference_only=.., learner_only=.., model_config=..) instead.

    # Create PPO configuration with default policies
    self.config = {
        "env": "network_env",
        "framework": "torch",
        "num_gpus": 0,
        "num_workers": 2,
        "train_batch_size": 4000,
        "sgd_minibatch_size": 128,
        "lr": 0.0003,
        "gamma": 0.99,
        "lambda": 0.95,
        "use_gae": True,
        "disable_env_checking": True,
        "_disable_preprocessor_api": True,
        "_disable_action_flattening": True,
        "experimental": {
            "_disable_rl_module": True,  # این خط را اضافه کنید
            "_validate_config": False,
            "enable_rl_module_and_learner": False,
            "enable_env_runner_and_connector_v2": False
        },  
        "clip_param": 0.2,
        "vf_loss_coeff": 1.0,
        "entropy_coeff": 0.0,
        "num_sgd_iter": 30,
        "rollout_fragment_length": 200,
        "batch_mode": "truncate_episodes",
        "model": {
            "fcnet_hiddens": [256, 256],
            "fcnet_activation": "tanh",
        },
        "multiagent": {
            "policies": {
                "orchestrator_policy": (None, env.orchestrator_observation_space, env.orchestrator_action_space, {}),
                "controller_policy": (None, env.controller_observation_space, env.controller_action_space, {})
            },
            "policy_mapping_fn": lambda agent_id, *args, **kwargs: (
                "orchestrator_policy" if agent_id.startswith("orchestrator")
                else "controller_policy"
            )
        },
        "observation_filter": "MeanStdFilter",
        "create_env_on_driver": False,
        "log_level": "INFO",
        # Disable experimental validation and new APIs
        "experimental": {
            "_validate_config": False,
            "enable_rl_module_and_learner": False,
            "enable_env_runner_and_connector_v2": False
        }
    }
    
    # Create trainer
    # Create trainer
    logger.info("Creating PPO trainer...")
    try:
        

        # Initialize trainer
        self.trainer = PPO(config=self.config)
        logger.info("PPO trainer created successfully")
    except Exception as e:
        logger.error(f"Error creating PPO trainer: {str(e)}")
        raise e

How can I address this problem???

2

Hello! Welcome to the Ray community :slight_smile:

The error you’re seeing happens because you’re using a deprecated method to initialize the RLModule in your PPO configuration. Based on the error message, you should replace RLModule(config=[RLModuleConfig]) with the new initialization format:

RLModule(
    observation_space=.., 
    action_space=.., 
    inference_only=.., 
    learner_only=.., 
    model_config=..
)

There’s a few diff ways you can resolve it but here are the basic steps:

  1. Update the RLModule Initialization – Make sure you’re passing the correct parameters when creating an RLModule. Instead of the old config argument, directly specify observation_space, action_space, and other necessary parameters.
  2. Enable the New API Stack – If you’re using an older configuration, migrate to the new API by enabling the RLModule and Learner APIs. You can do this by setting enable_rl_module_and_learner=True in your configuration.
  3. Check for Conflicting Settings – Double-check your config to ensure there are no conflicting parameters. For example, if _disable_rl_module=True is set in the experimental section, it could be causing issues when using RLModules.

TL;DR

  • Use the new RLModule API by specifying observation_space, action_space, and model_config directly.
  • Enable the RLModule and Learner APIs in your configuration to avoid compatibility issues.
  • Check for conflicting settings, such as _disable_rl_module=True, which may prevent RLModules from working correctly

Here’s some additional reading that might help:
Docs:

3

Hi Christina,
I face the same error when I tried to train the example:cartpole_with_dict_observation_space.py

would you mind explain more how it should be set the config and RLModule?
unfortunately I didn’t find an standard documents or a simple example for solving this problem! especially by new API stack.
my config is something as below :

config = (
    PPOConfig()
    .api_stack(
        enable_rl_module_and_learner=True,
        enable_env_runner_and_connector_v2=True,
    )
    .environment(
        env="CartPoleWithDictObservationSpace",
    )
    .training(
        lr = agent_params["lr"],  # Learning rate
        entropy_coeff= agent_params["entropy_coeff"],  # Encourage exploration with entropy regularization
    )
    .env_runners(num_env_runners = agent_params["num_env"])  # Number of parallel environments
    .framework("torch")
    
    .rl_module(
    rl_module_spec=RLModuleSpec(
        module_class=PPOTorchRLModule,
        inference_only=False,
        learner_only=False,
        observation_space=env_instance.observation_space,
        action_space=env_instance.action_space,
        model_config={"hidden_sizes": [128, 128]},  # Optional, passed to RLModule
    )
    )
   
)

thanks in advanced

1 Like
4

I also faced the below error when trying out, and found that the issue I had is because I was importing “override” from typing module. Instead if i use from ray’s library, it is working. Not sure if this is the right way and this is intentional

ray.rllib.utils.annotations import override

RLModule(config=[RLModuleConfig]) has been deprecated. Use RLModule(observation_space=.., action_space=.., inference_only=.., learner_only=.., model_config=..) instead.

5

Hi @kmprasad4u
interesting. Could you please share more info?
How you set your config?
Thanks

6

Edit: I see that you are using Ray’s PPOTorchRLModule and not a custom module. Then i think this may not help you. Ensure you have latest packages

This is not related to config. Below a snippet. There may be other reasons for the same error. Ensure you are using New API stack and passing the correct config as well


from typing import Any, Dict

from ray.rllib.algorithms.ppo import PPOConfig
from ray.rllib.core.rl_module.rl_module import RLModuleSpec
from ray.rllib.core.rl_module.torch.torch_rl_module import TorchRLModule
from ray.rllib.core.rl_module.apis.value_function_api import ValueFunctionAPI
from ray.rllib.utils.annotations import override

class NeuralNetworkModel(TorchRLModule, ValueFunctionAPI):
    def setup(self):
        # layer_dimensions = self.model_config["layer_dimensions"]
        pass

    # the override here, I was using "from typing". Changing it to from ray.rllib.utils.annotations resolved the issue for me. 
    @override(TorchRLModule)     
    def _forward(self, batch: Dict[str, Any], **kwargs) -> Dict[str, Any]:
        pass

config = (
    PPOConfig()
        .rl_module(
            rl_module_spec =  RLModuleSpec(
                module_class = NeuralNetworkModel,
                model_config={
                    # "layer_dimensions": (32, 16, 8)
                }
            )
        )
)
1 Like