How to select variables for optimizerd to exclude CNN from updates?

Halman · March 24, 2023, 2:21am

Medium: It contributes to significant difficulty to complete my task, but I can

I am trying to learn to use sac with image input. In this case, we are trying to implement a method in which Critic is updated including the convolutional network, and Actor shares the convolutional network with Critic and excludes the convolutional network from the parameters to be updated.

It seems that the parameters to be updated are determined using the following policy_variable function.

The original network is ComplexInputNetwork, and I think I can exclude the update parameters related to VisionNetwork from it, but I can’t think of a clever way to do it.

Can anyone give me a better way to do this?

github.com

ray-project/ray/blob/8a9a176a24280a48d63eba2bdd33245c123d46ca/rllib/algorithms/sac/sac_torch_model.py#L319


      
              if isinstance(model_out, dict) and "obs" in model_out:
                  # Model outs may come as original Tuple observations
                  if isinstance(self.action_model.obs_space, Box):
                      model_out["obs"] = concat_obs_if_necessary(model_out["obs"])
                  return self.action_model(model_out, state_in, seq_lens)
              else:
                  if isinstance(self.action_model.obs_space, Box):
                      model_out = concat_obs_if_necessary(model_out)
                  return self.action_model({"obs": model_out}, state_in, seq_lens)
          
          
def policy_variables(self):
              """Return the list of variables for the policy net."""
          
          
    return self.action_model.variables()
          
          
def q_variables(self):
              """Return the list of variables for Q / twin Q nets."""
          
          
    return self.q_net.variables() + (
                  self.twin_q_net.variables() if self.twin_q_net else []
              )

arturn · April 19, 2023, 6:44pm

Please have a look at the following thread: Partial freeze and partial train

Topic		Replies	Views
Is there no option to train SAC with a convolutional network? RLlib	3	358	August 21, 2021
Custome the SACTorchModel RLlib	2	22	December 10, 2024
What is the recommended way to make use of a trained model? RLlib	2	346	February 8, 2022
How to disable flattened Dict or Tuple observation in ComplexInputNetwork RLlib	5	524	July 9, 2021
Set model_config in RLlib RLlib	5	2088	February 24, 2021

How to select variables for optimizerd to exclude CNN from updates?

Related topics