Value Branch In fcnet.py

Zhao_Pengfei · September 12, 2021, 10:15am

In what kind of situations would I use the value_branch in FullyConnectedNetwork in fcnet.py ?

Since I am only using the forward pass, I don’t see any use of the value_branch. In which situations should it be used?

Also how can I bypass initializing model parameters for the value branch if I don’t need it ?

mannyv · September 12, 2021, 11:25am

It is primarily used by the policy gradient algorithms in the loss function and for computing the advantages on a sequence of episode steps.

If you do not have t need it you have two options.

The first is to ignore it if the only issue with having it there and not using it is wasting a little memory. If you use the default setting of "vf_share_layers": True there will be only 1 extra layer.

Your other option is to create your own custom model without a value branch/layer. Here are the documentation and example of a simple custom model.

https://docs.ray.io/en/latest/rllib-models.html#customizing-preprocessors-and-models

github.com

ray-project/ray/blob/master/rllib/examples/models/fast_model.py

from ray.rllib.models.modelv2 import ModelV2
from ray.rllib.models.tf.tf_modelv2 import TFModelV2
from ray.rllib.models.torch.misc import SlimFC
from ray.rllib.models.torch.torch_modelv2 import TorchModelV2
from ray.rllib.utils.annotations import override
from ray.rllib.utils.framework import try_import_tf, try_import_torch

tf1, tf, tfv = try_import_tf()
torch, nn = try_import_torch()


class FastModel(TFModelV2):
    """An example for a non-Keras ModelV2 in tf that learns a single weight.

    Defines all network architecture in `forward` (not `__init__` as it's
    usually done for Keras-style TFModelV2s).
    """

    def __init__(self, obs_space, action_space, num_outputs, model_config,
                 name):

This file has been truncated. show original

–

Topic		Replies	Views
What is the use for Model Layer "_value_branch" for Gradient Free Optimization (ES, ARS)? RLlib	1	244	September 27, 2021
[RLlib] why is TF value out initialization 100x smaller than torch? RLlib	3	286	January 27, 2021
Implementing a custom RNN using the TorchModelV2 RLlib	1	401	December 16, 2022
NN model for RLLib A3CPolicy RLlib	1	326	July 23, 2021
Variable-Sharing between policies RLlib	3	690	March 13, 2021

Value Branch In fcnet.py

Related Topics