Built in 2D Convolutions with LSTM

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

Hi all,

Just a quick question, if my input is 2D image and I configure my model as follows:

 "model": {

        "vf_share_layers": False,
        "use_lstm": True,
        "max_seq_len": 32,
        "lstm_cell_size": 128,
        "lstm_use_prev_action": True,

        "conv_filters":[
            [4, [3, 4], [1, 1]],
            [16, [6, 8], [3, 3]],
            [32, [6, 8], [3, 4]],
            [64, [6, 6], 3],
            [256, [9, 9], 1],
        ],

        "conv_activation": "relu",

        "post_fcnet_hiddens": [256, 256],
        "post_fcnet_activation": "relu",
    },

I just want to confirm that my understanding of the data flow is correct:

The input image is passed through the convolutions, then the post_fcnet_hiddens and then finaly into the LSTM cell?

If not, if someone could correct me that would be greatly appreciated!

Thanks!

2

Hey @Denys_Ashikhin ,

This API not not super transparent and this is on our radar. But yes, indeed.
You mental model of how things are stacked together is correct.

You can also print models nicely to find out how they look!
Here is a script of what experimenting with building models could look like:

import numpy as np
from ray.rllib.models.catalog import ModelCatalog
import gym

env = gym.make("CartPole-v1")

model = ModelCatalog.get_model_v2(
        obs_space=env.observation_space,
        action_space=env.action_space,
        num_outputs=np.array(1),
        framework="tf2",
        model_config={
        "vf_share_layers": False,
        "use_lstm": True,
        "max_seq_len": 32,
        "lstm_cell_size": 128,
        "lstm_use_prev_action": True,
        "fcnet_hiddens": [],
        "lstm_use_prev_reward": False,
        "conv_filters":[
            [4, [3, 4], [1, 1]],
            [16, [6, 8], [3, 3]],
            [32, [6, 8], [3, 4]],
            [64, [6, 6], 3],
            [256, [9, 9], 1],
        ],

        "conv_activation": "relu",

        "post_fcnet_hiddens": [256, 256],
        "post_fcnet_activation": "relu",
    },
    )

print(model._rnn_model.summary())

Cheers!

1 Like
3

Thank for clarifying that,

one final question: Is there any (non-custom model overwrite) way to insert FCNET(s) between the conv filters?

4

Hi @arturn ,
is it possible to also “print out or plot” the cnn model?
this:

print(model_cnn_summary())

didn’t work out

5

Hi @Seyar_Barez ,

You will have to add context to print(model_cnn_summary()).
All TF models have a printable representation.
But not the ModelV2-object itself - you will have to access the summary method of the TF model that is encapsulated by the ModelV2.

1 Like
6

@arturn

I was thinking this morning that perhaps the ModelV2 or or its current equivalent should have a sumarize method.

7

@mannyv Absolutely. I’ll open a feature request.

8

Thank you very much!
maybe something like this:
1st_Idea:
models_arr = analysis.get_model()
my_cnn = models_arr[0]
my_rnn = models_arr[1]
my_attention_net = models_arr[2]

2nd_Idea

analysis = tun.run(
“PPO”,
…
“plot_model”: True
)

plot_true would call :

tf.keras.utils.plot_model(

    #     self.base_model,
    #     to_file='/results/model.png',
    #     show_shapes=False,
    #     show_dtype=False,
    #     show_layer_names=True,
    #     rankdir='TB',
    #     expand_nested=False,
    #     dpi=96,
    #     layer_range=None,
    #     #show_layer_activations=False
    # )
1 Like