Requesting Clarification about Network Architectures Designs

Saurabh_Arora · July 8, 2021, 12:41pm

Hi all,

I request a clarification about network architectures used in rllib. In rllib, most algorithms (PPO, DQN, etc) are training a neural network approximation of a policy. Does that mean there is some relationship/ constraint between the total number of actions and the number of nodes in output layer?

arturn · July 14, 2021, 8:06am

Does that mean there is some relationship/ constraint between the total number of actions and the number of nodes in output layer?

Yes. The number of outputs relates to the actions through the action distribution.
In the link you can find multiple action distributions that RLlib uses and if you google them, you can see for each one how many output neurons a single action needs.

Saurabh_Arora · July 30, 2021, 10:49am

Thanks for response, @arturn .

For further clarification,

Ques 1) If I understood catalog code correctly, for discrete action space, the number of output nodes in policy network is same as the size of action space. Is that correct?

Ques 2) If not correct, does that mean the number of discrete space actions in MDP can be more than the number of output nodes of neural network approximating the policy?

@sven1977 , can you please help add information from your understanding?

mannyv · July 31, 2021, 11:10am

Hi @Saurabh_Arora,

Q1 is correct the number of outputs matches the number of possible actions.

github.com

ray-project/ray/blob/839fc592248a93e62715a65344d375cf47312833/rllib/models/tf/tf_action_dist.py#L99

    
      
                  p0 = ea0 / z0
                  return tf.reduce_sum(
                      p0 * (a0 - tf.math.log(z0) - a1 + tf.math.log(z1)), axis=1)
          
          
    @override(TFActionDistribution)
              def _build_sample_op(self) -> TensorType:
                  return tf.squeeze(tf.random.categorical(self.inputs, 1), axis=1)
          
          
    @staticmethod
              @override(ActionDistribution)
              def required_model_output_shape(action_space, model_config):
                  return action_space.n
          
          

          
class MultiCategorical(TFActionDistribution):
              """MultiCategorical distribution for MultiDiscrete action spaces."""
          
          
    def __init__(self,
                           inputs: List[TensorType],
                           model: ModelV2,
                           input_lens: Union[List[int], np.ndarray, Tuple[int, ...]],

github.com

ray-project/ray/blob/4cbe13cdfded336dbb814a7c10eba19075b9ba13/rllib/models/torch/torch_action_dist.py#L85

    
      
              @override(ActionDistribution)
              def deterministic_sample(self) -> TensorType:
                  self.last_sample = self.dist.probs.argmax(dim=1)
                  return self.last_sample
          
          
    @staticmethod
              @override(ActionDistribution)
              def required_model_output_shape(
                      action_space: gym.Space,
                      model_config: ModelConfigDict) -> Union[int, np.ndarray]:
                  return action_space.n
          
          

          
class TorchMultiCategorical(TorchDistributionWrapper):
              """MultiCategorical distribution for MultiDiscrete action spaces."""
          
          
    @override(TorchDistributionWrapper)
              def __init__(self,
                           inputs: List[TensorType],
                           model: TorchModelV2,
                           input_lens: Union[List[int], np.ndarray, Tuple[int, ...]],

Topic		Replies	Views
Next action in RLlib VisionNetworks RLlib	4	415	April 27, 2021
Examples of using multiple (simultaneous) actions? RLlib	1	437	February 12, 2021
Action space with multiple output? RLlib	7	923	July 14, 2022
My RLlib implementation seems to compute random actions RLlib	4	771	February 15, 2022
How to get DQN action distribution RLlib	2	330	November 3, 2022

Requesting Clarification about Network Architectures Designs

Related Topics