Built in 2D Convolutions with LSTM

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

Hi all,

Just a quick question, if my input is 2D image and I configure my model as follows:

 "model": {

        "vf_share_layers": False,
        "use_lstm": True,
        "max_seq_len": 32,
        "lstm_cell_size": 128,
        "lstm_use_prev_action": True,

        "conv_filters":[
            [4, [3, 4], [1, 1]],
            [16, [6, 8], [3, 3]],
            [32, [6, 8], [3, 4]],
            [64, [6, 6], 3],
            [256, [9, 9], 1],
        ],

        "conv_activation": "relu",

        "post_fcnet_hiddens": [256, 256],
        "post_fcnet_activation": "relu",
    },

I just want to confirm that my understanding of the data flow is correct:

The input image is passed through the convolutions, then the post_fcnet_hiddens and then finaly into the LSTM cell?

If not, if someone could correct me that would be greatly appreciated!

Thanks!

Hey @Denys_Ashikhin ,

This API not not super transparent and this is on our radar. But yes, indeed.
You mental model of how things are stacked together is correct.

You can also print models nicely to find out how they look!
Here is a script of what experimenting with building models could look like:

import numpy as np
from ray.rllib.models.catalog import ModelCatalog
import gym

env = gym.make("CartPole-v1")

model = ModelCatalog.get_model_v2(
        obs_space=env.observation_space,
        action_space=env.action_space,
        num_outputs=np.array(1),
        framework="tf2",
        model_config={
        "vf_share_layers": False,
        "use_lstm": True,
        "max_seq_len": 32,
        "lstm_cell_size": 128,
        "lstm_use_prev_action": True,
        "fcnet_hiddens": [],
        "lstm_use_prev_reward": False,
        "conv_filters":[
            [4, [3, 4], [1, 1]],
            [16, [6, 8], [3, 3]],
            [32, [6, 8], [3, 4]],
            [64, [6, 6], 3],
            [256, [9, 9], 1],
        ],

        "conv_activation": "relu",

        "post_fcnet_hiddens": [256, 256],
        "post_fcnet_activation": "relu",
    },
    )

print(model._rnn_model.summary())

Cheers!

Thank for clarifying that,

one final question: Is there any (non-custom model overwrite) way to insert FCNET(s) between the conv filters?