# Entropy value in IMPALA **URL:** \ **Category:** RLlib\ **Created:** [April 13, 2021, 8:25pm UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709 "2021-04-13T20:25:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Fabien-Couthouis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fabien-couthouis/32/814_2.png) [@Fabien-Couthouis](https://discuss.ray.io/u/Fabien-Couthouis)\ **Post date:** [April 13, 2021, 8:25pm UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/1 "2021-04-13T20:25:15Z")
Hello, I was wondering why entropy value is so high while training using IMPALA. Below is an example of the entropy values in IMPALA (blue) vs PPO (orange), and even on CartPole-v0, which only have an action space of size 2, entropy is above 250 on IMPALA. ![entropy](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/3c4db7047ac4f3b7abd8b1c88cd1ea40136999ff.png) I am using ray==1.2, tensorflow==2.3, with default hyperparameters on both PPO and IMPALA. I would appreciate any information about this. Thanks a lot! ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 14, 2021, 6:51am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/2 "2021-04-14T06:51:28Z")
Hey @Fabien-Couthouis , interesting observation. This cannot be correct, indeed :). The entropies should be the same (especially for cartpole after so many timesteps!). Will take a look. ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 14, 2021, 7:17am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/3 "2021-04-14T07:17:31Z")
Found it: For IMPALA, for some reason, we report the sum of all entropies over the batch (size 500 by default). For PPO, we report the mean. I’ll change IMPALA to also report the mean, instead. … ---
**Author:** ![Fabien-Couthouis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fabien-couthouis/32/814_2.png) [@Fabien-Couthouis](https://discuss.ray.io/u/Fabien-Couthouis)\ **Post date:** [April 14, 2021, 7:26am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/4 "2021-04-14T07:26:52Z")
Well it was quick! Thanks a lot @sven1977. ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 14, 2021, 7:32am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/5 "2021-04-14T07:32:15Z")
Here is the PR: [[RLlib] Discussion 1709: IMPALA (tf and torch) reports sum of entropy (over batch) in stats. Should report mean instead. by sven1977 · Pull Request #15290 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/15290) ---
**Author:** ![Fabien-Couthouis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fabien-couthouis/32/814_2.png) [@Fabien-Couthouis](https://discuss.ray.io/u/Fabien-Couthouis)\ **Post date:** [April 14, 2021, 11:10am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/6 "2021-04-14T11:10:00Z")
I think the same fix (report the mean instead of the sum) should also apply on pi\_loss and vf\_loss, as done in PPO, because the sum is a bit confusing. Do you agree? IMPALA (blue) vs PPO (orange) ![vf_loss](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/4755759f91853927480314f367f0b35fd56b5e55.png) (same with pi loss) ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 20, 2021, 9:21am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/7 "2021-04-20T09:21:55Z")
You are right. Could you do a PR with a fix for this? You can use the above PR as a template. > > > Discussion 1709: IMPALA (tf and torch) reports sum of entropy (over batch) in stats. Should report mean instead. > > Also see this discussion here: > https://discuss.ray.io/t/entropy-value-in-impala/1709 > > > \## Why are these changes needed? > > > \## Related issue number > > > \## Checks > > \- \[x\] I've run \`scripts/format.sh\` to lint the changes in this PR. > \- \[x\] I've included any doc changes needed for https://docs.ray.io/en/master/. > \- \[x\] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ > \- Testing Strategy > - \[x\] Unit tests > - \[\] Release tests > - \[\] This PR is not tested :( ---
**Author:** ![Fabien-Couthouis](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fabien-couthouis/32/814_2.png) [@Fabien-Couthouis](https://discuss.ray.io/u/Fabien-Couthouis)\ **Post date:** [April 20, 2021, 9:07pm UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/8 "2021-04-20T21:07:58Z")
The pull request can be found here: [[RLlib] Discussion 1709: IMPALA (tf and torch) reports sum of losses (over batch) in stats. Should report mean instead. by Fabien-Couthouis · Pull Request #15427 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/15427) ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 21, 2021, 8:59am UTC](https://discuss.ray.io/t/entropy-value-in-impala/1709/9 "2021-04-21T08:59:38Z")
Merged 🙂 Thanks for this quick fix @Fabien-Couthouis !