A few issues transitioning from Ray 2.3 to Ray 2.6.3 (Mutli-issues)

RLlib
1

Trying to overcome some issues I decided to upgrade to Ray 2.6.3 in hope of solving the issue.
I do not know if it is solved, but I definitely created new ones…

I’m getting some warnings, which I wish to understand and avoid.

So, I am using PPOConfig:

    from ray.rllib.algorithms.ppo import PPOConfig

    algo = (
        PPOConfig()
        .resources(
            num_gpus=1
            )
        .rollouts(
            num_rollout_workers=1,
            enable_connectors=True,
            batch_mode="truncate_episodes",
            )
        .framework(
            framework="torch",
            )
        .environment(
            env="gt-rtgym-env-v1",
            disable_env_checking=True,
            render_env=False,
            )
        
        .training(
            train_batch_size=155,
        )
        .build()
    )

Leads to:

2023-09-24 19:10:09,302	WARNING deprecation.py:50 -- DeprecationWarning: `DirectStepOptimizer` has been deprecated. This will raise an error in the future!
2023-09-24 19:10:09,767	WARNING algorithm_config.py:2558 -- Setting `exploration_config={}` because you set `_enable_rl_module_api=True`. When RLModule API are enabled, exploration_config can not be set. If you want to implement custom exploration behaviour, please modify the `forward_exploration` method of the RLModule at hand. On configs that have a default exploration config, this must be done with `config.exploration_config={}`.
2023-09-24 19:10:09,768	WARNING algorithm_config.py:2558 -- Setting `exploration_config={}` because you set `_enable_rl_module_api=True`. When RLModule API are enabled, exploration_config can not be set. If you want to implement custom exploration behaviour, please modify the `forward_exploration` method of the RLModule at hand. On configs that have a default exploration config, this must be done with `config.exploration_config={}`.
c:\Users\tom\anaconda3\envs\GTRay\lib\site-packages\ray\rllib\algorithms\algorithm.py:484: RayDeprecationWarning: This API is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable PYTHONWARNINGS="ignore::DeprecationWarning"
`UnifiedLogger` will be removed in Ray 2.7.
  return UnifiedLogger(config, logdir, loggers=None)
c:\Users\tom\anaconda3\envs\GTRay\lib\site-packages\ray\tune\logger\unified.py:53: RayDeprecationWarning: This API is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable PYTHONWARNINGS="ignore::DeprecationWarning"
The `JsonLogger interface is deprecated in favor of the `ray.tune.json.JsonLoggerCallback` interface and will be removed in Ray 2.7.
  self._loggers.append(cls(self.config, self.logdir, self.trial))
c:\Users\tom\anaconda3\envs\GTRay\lib\site-packages\ray\tune\logger\unified.py:53: RayDeprecationWarning: This API is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable PYTHONWARNINGS="ignore::DeprecationWarning"
The `CSVLogger interface is deprecated in favor of the `ray.tune.csv.CSVLoggerCallback` interface and will be removed in Ray 2.7.
  self._loggers.append(cls(self.config, self.logdir, self.trial))
c:\Users\tom\anaconda3\envs\GTRay\lib\site-packages\ray\tune\logger\unified.py:53: RayDeprecationWarning: This API is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable PYTHONWARNINGS="ignore::DeprecationWarning"
The `TBXLogger interface is deprecated in favor of the `ray.tune.tensorboardx.TBXLoggerCallback` interface and will be removed in Ray 2.7.
  self._loggers.append(cls(self.config, self.logdir, self.trial))
(pid=19280) DeprecationWarning: `DirectStepOptimizer` has been deprecated. This will raise an error in the future!

So some questions there… where have I set _enable_rl_module_api=True?

2

I might also wanna understand the details of this sort of warnings