Set model_config in RLlib

RLlib
1

Hello Guys, does anyone knows how to change the ConvNet configuration of a keras model in Rllib? I have a 10101 observation matrix and I would like to use it as an image for the ConvNet. but I got the following error. I know that I need to modify the config, but still do not know how to do that? does anyone has an example?
ValueError: No default configuration for obs shape [10, 10, 1], you must specify conv_filters manually as a model option. Default configurations are only available for inputs of shape [42, 42, K] and [84, 84, K]. You may alternatively want to use a custom model or preprocessor.

1 Like
2

Hello, if you are using Tune with RLLib, you can include your changes in the config parameter, like in this example.

2 Likes
3

Hello, thanks for your reply. no I’m not using Tune for now.
i defined the config, but it does not work. this is my code. my observation space is 10x10x1.

import os
import gym
import ray
import tensorflow as tf
from ray.rllib.agents import dqn 
# from ray.rllib.agents.dqn import DQN_CONFIG
from ray.rllib.models.tf.tf_modelv2 import TFModelV2
from ray.rllib.models.catalog import ModelCatalog, MODEL_DEFAULTS
from ray.rllib.models.tf.misc import normc_initializer
from ray.tune.logger import pretty_print
from environment import Environment


class CustomModel(TFModelV2):
    def __init__(self, obs_space, action_space, num_outputs, model_config,
                 name):
        super(CustomModel, self).__init__(obs_space, action_space, num_outputs,
                                          model_config, name)
        input_layer = tf.keras.layers.Input(shape=obs_space.shape, name="observations")
        hidden_layer = tf.keras.layers.Dense(128, name="hidden_layer", activation=tf.nn.relu, 
                            kernel_initializer=normc_initializer(1.0))(input_layer)
        output_layer = tf.keras.layers.Dense(num_outputs, name="my_out", activation=None, 
                            kernel_initializer=normc_initializer(0.01))(hidden_layer)
        value_layer = tf.keras.layers.Dense(1, name="value_out", activation=None,
                            kernel_initializer=normc_initializer(0.01))(hidden_layer)
        self.base_model = tf.keras.Model(
            input_layer, [output_layer, value_layer])
        self.register_variables(self.base_model.variables)
    
    
    def forward(self, input_dict, state, seq_lens):
        return self.model.forward(input_dict, state, seq_lens)

    def value_function(self):
        return self.model.value_function()

def train_agent(agent, num_steps):
    for i in range(num_steps):
        result = agent.train()
        if i%20 == 0:
            print(pretty_print(result))
            
if __name__ == "__main__":
    ray.init(ignore_reinit_error=True)
    ModelCatalog.register_custom_model(
        "my_model", CustomModel)
        
    config = {
        "env": Environment,
        "env_config": {},
        
        "model": {
            "custom_model": "my_model",
            "dim": 10, 
            "grayscale": True,
            "conv_filters": [[16, [4, 4], 2], [32, [4, 4], 2], [512, [11, 11], 1]],
        },
        
        "num_gpus": int(os.environ.get("RLLIB_NUM_GPUS", "1")),
        "num_workers": 1,  # parallelism
    }
    
    
    agent = dqn.DQNTrainer(env=Environment, config=config)
    
    train_agent(agent, 1000)
    
    ray.shutdown()
4

Thanks for posting the question @deepgravity .

Some thoughts:

  • Settings inside the model config dict (except “custom_model” and “custom_model_config”!) are usually only for configuring RLlib’s default models. E.g. fcnet_hiddens or conv_filters are only used by our default models. Once you define a custom model, these keys are ignored (unless, you specifically utilize them in your model’s c’tor via e.g. hiddens = model_config["fcnet_hiddens"] ...).

  • You don’t seem to have any Conv2D layers in your custom model so it won’t be able to process images properly.

  • Your custom model seems simple enough for me to suggest just using our default models (VisionNet). This way you can actually configure it using the conv_filters key.

  • Valid conv_filters would be e.g.: [[16, [4, 4], 2], [32, [4, 4], 2], [512, [2, 2], 2]], but you should try other conv settings as well. Remember to remove your “custom_model” from the config, otherwise, there will be no Conv2D layers at all.

1 Like
5

many thanks. I think i need to work on my code more! Is there any example in the repo which uses visionnet?

1 Like
6

Cool. Yes, we have examples (even though not many :confused: ) , e.g. there is a PPO test case. which runs on an Atari env:
rllib/agents/ppo/tests/test_ppo.py::TestPPO::test_ppo_compilation_and_lr_schedule

In there, you can just reduce the loop to only run the “MsPacmanNoFrameskip-v4” env and w/o the LSTM wrapping. If you set local_mode=True inside the ray.init(local_mode=True) call further above in the same python file, you should be able to debug the creation of the default VisionNet that will be built then.