How can I check the decrease in loss during the study?

Halman · June 7, 2023, 1:29am

Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I am using ver 2.2 of ray rllib.
I am training using the SAC algorithm and would like to check the changes in Actor loss and Critic loss to see if the training is progressing properly.

In the file progress.csv, which is output without any modification, the value of reward is output, but the loss is not.

My question is as follows.

Are the changes in loss output to some log file when we start the study with the default settings?
What code change should I make to output the loss to progress.csv? I would like to know the specific method.

kourosh · June 7, 2023, 6:52pm

You can open tensorboard (or wandb if you have enabled it) and look for “actor_loss” or “critic_loss”. They should already be there (at least on the latest version on master)

github.com

ray-project/ray/blob/master/rllib/algorithms/sac/sac_torch_policy.py#L376-L379


      
          "actor_loss": torch.mean(torch.stack(policy.get_tower_stats("actor_loss"))),
          "critic_loss": torch.mean(
              torch.stack(tree.flatten(policy.get_tower_stats("critic_loss")))
          ),

Halman · June 12, 2023, 6:04am

Thanks! It works for me.