ValueError: `RLModule(config=[RLModuleConfig])` has been deprecated

RLlib
1

Hello,

I encounter this error while passing the config to PPO:
(MultiAgentEnvRunner pid=28036) File “D:\Miniconda\envs\condaPars\Lib\site-packages\ray\rllib\core\rl_module\rl_module.py”, line 113, in build
(MultiAgentEnvRunner pid=28036) module = self.module_class(module_config)
(MultiAgentEnvRunner pid=28036) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(MultiAgentEnvRunner pid=28036) deprecation_warning(
(MultiAgentEnvRunner pid=28036) File “D:\Miniconda\envs\condaPars\Lib\site-packages\ray\rllib\utils\deprecation.py”, line 48, in deprecation_warning
(MultiAgentEnvRunner pid=28036) raise ValueError(msg)
(MultiAgentEnvRunner pid=28036) ValueError: RLModule(config=[RLModuleConfig]) has been deprecated. Use RLModule(observation_space=.., action_space=.., inference_only=.., learner_only=.., model_config=..) instead.

    # Create PPO configuration with default policies
    self.config = {
        "env": "network_env",
        "framework": "torch",
        "num_gpus": 0,
        "num_workers": 2,
        "train_batch_size": 4000,
        "sgd_minibatch_size": 128,
        "lr": 0.0003,
        "gamma": 0.99,
        "lambda": 0.95,
        "use_gae": True,
        "disable_env_checking": True,
        "_disable_preprocessor_api": True,
        "_disable_action_flattening": True,
        "experimental": {
            "_disable_rl_module": True,  # این خط را اضافه کنید
            "_validate_config": False,
            "enable_rl_module_and_learner": False,
            "enable_env_runner_and_connector_v2": False
        },  
        "clip_param": 0.2,
        "vf_loss_coeff": 1.0,
        "entropy_coeff": 0.0,
        "num_sgd_iter": 30,
        "rollout_fragment_length": 200,
        "batch_mode": "truncate_episodes",
        "model": {
            "fcnet_hiddens": [256, 256],
            "fcnet_activation": "tanh",
        },
        "multiagent": {
            "policies": {
                "orchestrator_policy": (None, env.orchestrator_observation_space, env.orchestrator_action_space, {}),
                "controller_policy": (None, env.controller_observation_space, env.controller_action_space, {})
            },
            "policy_mapping_fn": lambda agent_id, *args, **kwargs: (
                "orchestrator_policy" if agent_id.startswith("orchestrator")
                else "controller_policy"
            )
        },
        "observation_filter": "MeanStdFilter",
        "create_env_on_driver": False,
        "log_level": "INFO",
        # Disable experimental validation and new APIs
        "experimental": {
            "_validate_config": False,
            "enable_rl_module_and_learner": False,
            "enable_env_runner_and_connector_v2": False
        }
    }
    
    # Create trainer
    # Create trainer
    logger.info("Creating PPO trainer...")
    try:
        

        # Initialize trainer
        self.trainer = PPO(config=self.config)
        logger.info("PPO trainer created successfully")
    except Exception as e:
        logger.error(f"Error creating PPO trainer: {str(e)}")
        raise e

How can I address this problem???

2

Hello! Welcome to the Ray community :slight_smile:

The error you’re seeing happens because you’re using a deprecated method to initialize the RLModule in your PPO configuration. Based on the error message, you should replace RLModule(config=[RLModuleConfig]) with the new initialization format:

RLModule(
    observation_space=.., 
    action_space=.., 
    inference_only=.., 
    learner_only=.., 
    model_config=..
)

There’s a few diff ways you can resolve it but here are the basic steps:

  1. Update the RLModule Initialization – Make sure you’re passing the correct parameters when creating an RLModule. Instead of the old config argument, directly specify observation_space, action_space, and other necessary parameters.
  2. Enable the New API Stack – If you’re using an older configuration, migrate to the new API by enabling the RLModule and Learner APIs. You can do this by setting enable_rl_module_and_learner=True in your configuration.
  3. Check for Conflicting Settings – Double-check your config to ensure there are no conflicting parameters. For example, if _disable_rl_module=True is set in the experimental section, it could be causing issues when using RLModules.

TL;DR

  • Use the new RLModule API by specifying observation_space, action_space, and model_config directly.
  • Enable the RLModule and Learner APIs in your configuration to avoid compatibility issues.
  • Check for conflicting settings, such as _disable_rl_module=True, which may prevent RLModules from working correctly

Here’s some additional reading that might help:
Docs:

3

Hi Christina,
I face the same error when I tried to train the example:cartpole_with_dict_observation_space.py

would you mind explain more how it should be set the config and RLModule?
unfortunately I didn’t find an standard documents or a simple example for solving this problem! especially by new API stack.
my config is something as below :

config = (
    PPOConfig()
    .api_stack(
        enable_rl_module_and_learner=True,
        enable_env_runner_and_connector_v2=True,
    )
    .environment(
        env="CartPoleWithDictObservationSpace",
    )
    .training(
        lr = agent_params["lr"],  # Learning rate
        entropy_coeff= agent_params["entropy_coeff"],  # Encourage exploration with entropy regularization
    )
    .env_runners(num_env_runners = agent_params["num_env"])  # Number of parallel environments
    .framework("torch")
    
    .rl_module(
    rl_module_spec=RLModuleSpec(
        module_class=PPOTorchRLModule,
        inference_only=False,
        learner_only=False,
        observation_space=env_instance.observation_space,
        action_space=env_instance.action_space,
        model_config={"hidden_sizes": [128, 128]},  # Optional, passed to RLModule
    )
    )
   
)

thanks in advanced

1 Like