[BUG] Heavy logic problem in validate_config for R2D2

LukasNothhelfer · December 12, 2021, 6:12pm

Shouldn’t a function that calls itself validate_config operate passively on the data and simply provide feedback as to whether the configuration is valid or not? The problem is currently in validate_config for R2D2 where the data to be validated is changed so that two successive calls result in a crash on the second call.

github.com

ray-project/ray/blob/f1acabe9cf37d5d123017fb3f158c37fb36499a5/rllib/agents/dqn/r2d2.py#L116

    
      
          
          
@override(DQNTrainer)
          def validate_config(self, config: TrainerConfigDict) -> None:
              """Checks and updates the config based on settings.
          
          
    Rewrites rollout_fragment_length to take into account burn-in and
              max_seq_len truncation.
              """
              super().validate_config(config)
          
          
    if config["replay_sequence_length"] != -1:
                  raise ValueError(
                      "`replay_sequence_length` is calculated automatically to be "
                      "model->max_seq_len + burn_in!")
              # Add the `burn_in` to the Model's max_seq_len.
              # Set the replay sequence length to the max_seq_len of the model.
              config["replay_sequence_length"] = \
                  config["burn_in"] + config["model"]["max_seq_len"]
          
          
    if config.get("batch_mode") != "complete_episodes":
                  raise ValueError("`batch_mode` must be 'complete_episodes'!")

Lets say you have a config which is valid and you do:
validate_config(config)
This will alter the config so a second call
validate_config(config)
will cause a crash.

Minimum example:

import ray.rllib.agents.dqn as dqn
config = dqn.r2d2.DEFAULT_CONFIG.copy()
dqn.r2d2.validate_config(config)
dqn.r2d2.validate_config(config) ### This will cause the crash

Unfortunately, it is not currently possible to use R2D2 with the Tune API. For unknown reasons, Tune (or whoever) calls the validate_config function multiple times when you start Tune as follows:

tune.run(
    "R2D2",
    config=config,
    ...
)

Topic		Replies	Views
Cannot create R2D2 trainer with evaluation worker RLlib	6	635	March 8, 2022
Reproduce R2D2 paper RLlib	2	463	June 28, 2022
Trouble reproducing results with DQN RLlib	3	430	April 14, 2023
Crash during training: paramter logits has invalid values	3	1017	August 9, 2021
Can't pass over custom_model_config to custom_model RLlib	3	436	March 9, 2023

[BUG] Heavy logic problem in validate_config for R2D2

Related topics