I only skimmed trough your topic. If it is the same warning message I am aware of you can safely ignore it. Currently there will always be a deprecation warning for the very first RLModuleConfig config created.
There still exists deprecated code that creates a self.config = RLModuleConfig(... from your new stack options for backwards compatibility, this instantiation throws the error
I wrote myself a snippet to run before my code to suppress this misleading warning
from ray.rllib.utils.deprecation import logger as __deprecation_logger
# This suppresses a deprecation warning from RLModuleConfig
__old_level = __deprecation_logger.getEffectiveLevel()
__deprecation_logger.setLevel(logging.ERROR)
RLModuleConfig()
__deprecation_logger.setLevel(__old_level)
del __deprecation_logger
Try if that hides the error. To be sure run it with a debugger and find the location, it could be a false postive and unrelated to your code.