# Intended workflow with for custom models using num\_outputs

**URL:** <https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058>\
**Category:** RLlib\
**Created:** [May 5, 2021, 9:28am UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058 "2021-05-05T09:28:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vakker00](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vakker00](https://discuss.ray.io/u/vakker00)\
**Post date:** [May 5, 2021, 9:28am UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058/1 "2021-05-05T09:28:12Z")

</div>

What’s the correct way to handle `num_outputs` in a custom model?

1. it seems that the action space should be used to get the correct output size, so `num_outputs` seems redundant in that case.
2. It’s not passed in to the model initializer in some cases (and the last layer is just the flattened features in the examples), so I’m wondering what’s the logic behind that and in what scenarios the `if num_outputs` check is needed.
3. Wrappers change `num_outputs` in a somewhat confusing way, but I assume the `num_outputs` of the wrapper should be separate from the wrapped class’ one, e.g. from `LSTMWrapper`:

```python
        # Add prev-action/reward nodes to input to LSTM.
        if self.use_prev_action:
            self.num_outputs += self.action_dim
        if self.use_prev_reward:
            self.num_outputs += 1

        self.lstm = nn.LSTM(
            self.num_outputs, self.cell_size, batch_first=not self.time_major)

        self.num_outputs = num_outputs

```

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [May 5, 2021, 10:59pm UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058/2 "2021-05-05T22:59:32Z")

</div>

Hi!

1. Usually we use the outputs of a model to parameterize a probability distribution of actions, with one neuron representing the mean and another representing the the variance or standard deviation of one gaussian distribution per action. So the connection between action\_space and num\_outputs is clear. But you can wrap models and depending on how you do so, it makes sense to me that they are both passed to many models. The `LSTMWrapper` that you talk about under (3) looks like good example for that to me.
2. As you said under (1), in many cases the information is redundant (but in others its not?).
3. The [LSTMWrapper](https://github.com/ray-project/ray/blob/master/rllib/models/tf/recurrent_net.py) uses num\_outputs in a way that is a little confusing:

.

```
# At this point, self.num_outputs is the number of nodes coming
# from the wrapped (underlying) model. In other words, self.num_outputs
# is the input size for the LSTM layer

```

First, num\_outputs is used by the wrapped model and holds the number of its outputs.

```
# Add prev-action/reward nodes to input to LSTM.
if self.use_prev_action:
    self.num_outputs += self.action_dim
if self.use_prev_reward:
    self.num_outputs += 1

```

Then, we use num\_outputs to add up how many input neurons the LSTM layer should have. This is where it gets a little confusing, since at times the outputs of this model never become what num\_outputs is.

```
# Set self.num_outputs to the number of output nodes desired by the
# caller of this constructor.
self.num_outputs = num_outputs

```

In the end num\_outputs becomes what we expect it to be.

So in the simplest case, the underlying model would want to know only one, action or observations space, because one conditions the other. Some models do not care about the action\_space and simply provide num\_outputs output neurons because they want RLlib to attach the last layer, depending on the action sampling. But the `LSTMWrapper` needs to know the action\_space and does not care about num\_outputs. So in order to support both, you can pass both to the model API.

This is how I understand the code. Does it make sense to you?

---

<div class="post-metadata">

**Author:** ![vakker00](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vakker00](https://discuss.ray.io/u/vakker00)\
**Post date:** [May 6, 2021, 8:56am UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058/3 "2021-05-06T08:56:30Z")

</div>

Thanks @arturn for the explanation.

* * *

> Usually we use the outputs of a model to parameterize a probability distribution of actions, with one neuron representing the mean and another representing the the variance or standard deviation of one gaussian distribution per action

I think this greatly depends on the action distribution, e.g. see [here](https://github.com/ray-project/ray/blob/e973b726c2e4039bd5a35ecfffad5028a15cdfee/rllib/models/catalog.py#L194), the one that you mentioned is the `# Box space -> DiagGaussian OR Deterministic.` case if I’m not mistaken.

* * *

> But the `LSTMWrapper` needs to know the action\_space and does not care about num\_outputs.

I would assume that because of `self.num_outputs += self.action_dim` it does need both.

* * *

Digging a bit more into this, I think the main difference can be seen [here](https://github.com/ray-project/ray/blob/4cbe13cdfded336dbb814a7c10eba19075b9ba13/rllib/policy/policy_template.py#L227), where the policy classes are being instantiated.

What I find weird is that if the model is customized then it doesn’t pass the `num_outputs` to the model’s constructor:

```python
                self.model = make_model(self, obs_space, action_space, config)
                dist_class, _ = ModelCatalog.get_action_dist(
                    action_space, self.config["model"], framework=framework)

```

I would expect (similarly to the custom dist cases) something like this:

```python
                dist_class, num_outputs = ModelCatalog.get_action_dist(
                    action_space, self.config["model"], framework=framework)
                self.model = make_model(self, obs_space, action_space, num_outputs=num_outputs, config)

```

This seems to imply that the `num_outputs` needs to be handled explicitly inside a custom policy’s constructor.

Maybe @sven1977 can shed some light on if this was intended and why? Or I’m just missing something obvious.

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [May 6, 2021, 9:32am UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058/4 "2021-05-06T09:32:19Z")

</div>

> I think this greatly depends on the action distribution, e.g. see [here](https://github.com/ray-project/ray/blob/e973b726c2e4039bd5a35ecfffad5028a15cdfee/rllib/models/catalog.py#L194), the one that you mentioned is the `# Box space -> DiagGaussian OR Deterministic.` case if I’m not mistaken.

Yes, I was indeed referring to the DiagGaussian, which I think is chosen by RLLib by default. Of course there are others!

> I would assume that because of `self.num_outputs += self.action_dim` it does need both.

Your are right, it does require both but not until further downwards. `self.num_outputs` depends on the wrapped model in the line that you reference. Further down, the `num_outptus` that is passed to the LSTMWrapper’s \ __init\__ is actually used. But since it requires both, num\_outputs and action\_space, this still answers (1).

I am not that certain regarding your last point.

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [May 7, 2021, 6:37pm UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058/5 "2021-05-07T18:37:58Z")

</div>

Hi @vakker00,

I think two conceptually different things are being lumped together. There are two types of customization. The most common type of customization is defining a new neural network architecture to use as your model. In this case the architecture is customized but the way rllib builds that model is not.

The second type of customization is how rllib builds the model. The `make_model` kwarg is a callable that allows you to customize that process. A quick look [at this example](https://github.com/ray-project/ray/blob/35ec91c4e04c67adc7123aa8461cf50923a316b4/rllib/agents/dqn/simple_q_tf_policy.py#L60) will show you how that is used. If you look inside they are actually using the standard way to build two different networks and do in fact use num\_outputs there.

num\_outputs is a variable that makes it easy to wrap one model in another. So in the case of an LSTMWrapper, self.num\_outputs actually means the number of outputs coming out of the layer that is being automatically wrapped. It we have a FC network that is being wrapped then its final layer is not the ultimate output layer it is an input layer to the LSTM and the LSTMwrapper with have the final output layer. In that case self.num\_outputs is expressing the number of units in the last layer before the lstm and the variable passed in as `num_outputs` will be the number of values returned by a call to the LSTMwrappers forward method. So member variable is the input size and function argument is the output size.

The reason that the action\_distribution does not take a num\_outputs argument is because the action\_distributions used in rllib are entirely defined by the shapes of the input and the action space when they are constructed.

---

<div class="post-metadata">

**Author:** ![vakker00](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vakker00](https://discuss.ray.io/u/vakker00)\
**Post date:** [May 10, 2021, 4:22pm UTC](https://discuss.ray.io/t/intended-workflow-with-for-custom-models-using-num-outputs/2058/6 "2021-05-10T16:22:18Z")

</div>

@mannyv thanks for the clarification, it makes sense.

> The reason that the action\_distribution does not take a num\_outputs argument is because the action\_distributions used in rllib are entirely defined by the shapes of the input and the action space when they are constructed.

The only thing I would add to this, is that it’s not clear why [here](https://github.com/ray-project/ray/blob/4cbe13cdfded336dbb814a7c10eba19075b9ba13/rllib/policy/policy_template.py#L227) the action\_distributions’ calculated num\_outputs is not used in make\_model, not the other way around. I would make sense if it was the the same way how it’s implemented a couple of lines below ([here](https://github.com/ray-project/ray/blob/4cbe13cdfded336dbb814a7c10eba19075b9ba13/rllib/policy/policy_template.py#L239)).