PPO nan in actor logits

RLlib
7

@tlaurie thanks for your post. @mannyv 's guess is as usual excellent and points directly to the problem. I do not want to get into the details here as @mannyv and you have already brought them up.

I would like to point to two options in RLlib that implement the suggestions given in this thread.

  1. The log standard deviation trained as a simple nn.Parameter (in torch) was already implemented (I admit it was not well documented): using in the model_config_dict {"free_log_std": True} will use the TorchFreeLogStdMLPHead that optimizes the standard deviation in form of a freely moving bias term.
  2. As a response to this thread here, we also added log-std clipping in similar form as shown above to all algorithms with continuous actions. Using in your model_config_dict {"log_std_clip_param": 1} would clamp the log-stds of all actions in between [-1,1] (the default is 20 and if you want to avoid clipping use float("inf")).

Thanks again to all who have contributed to this discussion and helped thereby improving this library!