Adding multiple layers on top of DQN Model

RLlib
1

Hi all,

I’m looking to use a GNN to preprocess some information before sending it into a DQN network.
I was wondering how I best connect these layers to the model generated by RLLib.
I prefer on using the RLLib generated models, as they are tried and true.

I found the templates available for custom models (TorchModelV2), but they demand complete custom implementations. I also discovered that they build upon the FullyConnectedNetwork, but this seems the wrong approach to append layers.

Any insight would be greatly appreciated!

Kind regards,

2

Hey @Vesyrak , the best way to customize the DQN model is to simply sub-class the existing ones and add new layers into these as required.

You can plug in your custom DQN model via:

config:
  model:
    custom_model: [your new sub-class of `DQNTorch|TFModel`]
    custom_model_config:
      [some c'tor args for your model class]

A simpler mechanism is maybe to use the already existing hiddens config parameter of DQN. This allows you to specify layer sizes after(!) the Advantages(A)/Value(V)-split. Hence, each of the A- and V-branches will have this structure of Dense layers. To customize the NN before this A/V-split, use config["model"]["fcnet_hiddens"].