# Requesting Clarification about Network Architectures Designs

**URL:** <https://discuss.ray.io/t/requesting-clarification-about-network-architectures-designs/2826>\
**Category:** RLlib\
**Created:** [July 8, 2021, 12:41pm UTC](https://discuss.ray.io/t/requesting-clarification-about-network-architectures-designs/2826 "2021-07-08T12:41:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saurabh\_Arora](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/saurabh_arora/32/1012_2.png) [@Saurabh\_Arora](https://discuss.ray.io/u/Saurabh_Arora)\
**Post date:** [July 8, 2021, 12:41pm UTC](https://discuss.ray.io/t/requesting-clarification-about-network-architectures-designs/2826/1 "2021-07-08T12:41:37Z")

</div>

Hi all,

I request a clarification about network architectures used in rllib. In rllib, most algorithms (PPO, DQN, etc) are training a neural network approximation of a policy. Does that mean there is some relationship/ constraint between the total number of actions and the number of nodes in output layer?

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [July 14, 2021, 8:06am UTC](https://discuss.ray.io/t/requesting-clarification-about-network-architectures-designs/2826/2 "2021-07-14T08:06:31Z")

</div>

> Does that mean there is some relationship/ constraint between the total number of actions and the number of nodes in output layer?

Yes. The number of outputs relates to the actions through the [action distribution](https://github.com/ray-project/ray/blob/e973b726c2e4039bd5a35ecfffad5028a15cdfee/rllib/models/catalog.py#L194).  
In the link you can find multiple action distributions that RLlib uses and if you google them, you can see for each one how many output neurons a single action needs.

---

<div class="post-metadata">

**Author:** ![Saurabh\_Arora](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/saurabh_arora/32/1012_2.png) [@Saurabh\_Arora](https://discuss.ray.io/u/Saurabh_Arora)\
**Post date:** [July 30, 2021, 10:49am UTC](https://discuss.ray.io/t/requesting-clarification-about-network-architectures-designs/2826/3 "2021-07-30T10:49:57Z")

</div>

Thanks for response, @arturn .

For further clarification,

Ques 1) If I understood catalog code correctly, for discrete action space, the number of output nodes in policy network is same as the size of action space. Is that correct?

Ques 2) If not correct, does that mean the number of discrete space actions in MDP can be more than the number of output nodes of neural network approximating the policy?

@sven1977 , can you please help add information from your understanding?

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [July 31, 2021, 11:10am UTC](https://discuss.ray.io/t/requesting-clarification-about-network-architectures-designs/2826/4 "2021-07-31T11:10:09Z")

</div>

Hi @Saurabh_Arora,

Q1 is correct the number of outputs matches the number of possible actions.

> <https://github.com/ray-project/ray/blob/839fc592248a93e62715a65344d375cf47312833/rllib/models/tf/tf_action_dist.py#L99>

> <https://github.com/ray-project/ray/blob/4cbe13cdfded336dbb814a7c10eba19075b9ba13/rllib/models/torch/torch_action_dist.py#L85>