Hello,
I was wondering why entropy value is so high while training using IMPALA. Below is an example of the entropy values in IMPALA (blue) vs PPO (orange), and even on CartPole-v0, which only have an action space of size 2, entropy is above 250 on IMPALA.
I am using ray==1.2, tensorflow==2.3, with default hyperparameters on both PPO and IMPALA.
I would appreciate any information about this. Thanks a lot!
Hey @Fabien-Couthouis , interesting observation. This cannot be correct, indeed :).
The entropies should be the same (especially for cartpole after so many timesteps!). Will take a look.
Found it: For IMPALA, for some reason, we report the sum of all entropies over the batch (size 500 by default). For PPO, we report the mean.
I’ll change IMPALA to also report the mean, instead. …
I think the same fix (report the mean instead of the sum) should also apply on pi_loss and vf_loss, as done in PPO, because the sum is a bit confusing.
Do you agree?
IMPALA (blue) vs PPO (orange)
(same with pi loss)
You are right. Could you do a PR with a fix for this? You can use the above PR as a template.
