# Does ReLU activation interfere with DQN?

**URL:** <https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537>\
**Category:** RLlib\
**Created:** [March 24, 2022, 10:17am UTC](https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537 "2022-03-24T10:17:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fedetask](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fedetask/32/2233_2.png) [@fedetask](https://discuss.ray.io/u/fedetask)\
**Post date:** [March 24, 2022, 10:17am UTC](https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537/1 "2022-03-24T10:17:12Z")

</div>

**How severe does this issue affect your experience of using Ray?** Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I have an environment where all rewards are negative and I’m using the rllib DQN implementation. In the model config, I’m setting `'fcnet_activation' = 'relu'`. What I would expect is that since all rewards are negative, the Q values would not be able to converge since the ReLU activation function does not allow for negative outputs.

However, the model seems to work just fine.

To investigate into this more, I tried adding `'post_fcnet_hiddens': [256]` and `'post_fcnet_activation': tune.grid_search(None, 'relu')`. Again I would expect the algorithm to work better with `None` rather than with `relu`, but the two seem to work equally good.

Is RLlib automatically modifying the activation of the last layer when DQN is used? Or both `'fcnet_activation'` and `'post_fcnet_activation'` are not applied to the final layer outputs from which DQN gets the Q values?

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [March 24, 2022, 10:08pm UTC](https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537/2 "2022-03-24T22:08:16Z")

</div>

Hi @fedetask ,

as far as I can only guess here, you use the default DQN - not touching most of its configuration parameters especially the ones it inherits silently from the `MODEL_DEFAULTS`. The DQN agent build its networks (Q- and Target-network) by calling for each [`ModelCatalog.get_model_v2()`](https://github.com/ray-project/ray/blob/d39cef572596be8eba72a01ef48d3df5244df729/rllib/agents/dqn/dqn_tf_policy.py#L180) (with `num_outputs` - becomes important shortly).  
Calling this function activates RLlib’s semi-automatic model choice dependent on input and output shape (`obs` and `num_outputs`). If you look then into the different default model classes you will see something like this:

```auto
if num_outputs:
      logits_out = tf.keras.layers.Dense(
          num_outputs,
          name="fc_out",
          activation=None,
          kernel_initializer=normc_initializer(0.01),
      )(last_layer)

```

i.e. the last layer is always `None`-activated. Now, I do not know how your observation space looks like, but you see will this in many different default networks (Fully Connected Network, CNN, etc.).

For the hidden layers the negative rewards do not play a role as you do not feed these rewards into the network - and even if: networks get randomly initialized such that the layer outcomes should not be all negative. Then the last layer is `None`-activated and negative output values can occur.

Hope his helps.  
Simon

---

<div class="post-metadata">

**Author:** ![fedetask](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fedetask/32/2233_2.png) [@fedetask](https://discuss.ray.io/u/fedetask)\
**Post date:** [March 28, 2022, 10:12am UTC](https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537/3 "2022-03-28T10:12:57Z")

</div>

It does help, thank you very much! So not even `post_fcnet_hiddens` and `post_fcnet_activation` modify this behavior, right?

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [March 28, 2022, 9:32pm UTC](https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537/4 "2022-03-28T21:32:28Z")

</div>

@fedetask, exactly. The layer with a `None`activation is always attached to the network. If you need another layer at the end, you would need to write a custom model.

---

<div class="post-metadata">

**Author:** ![fedetask](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fedetask/32/2233_2.png) [@fedetask](https://discuss.ray.io/u/fedetask)\
**Post date:** [April 29, 2022, 12:33pm UTC](https://discuss.ray.io/t/does-relu-activation-interfere-with-dqn/5537/5 "2022-04-29T12:33:01Z")

</div>

Hello, sorry if I still bother you

I am using a custom model, but that final layer is still added somewhere by RLlib. I want to disable this behavior to handle q values directly in my custom model (I need to implement action masking so I need the output of my model to already be the masked Q values). How can I do this?