# Add episode reward variance into matrix and tensorboard

**URL:** <https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038>\
**Category:** RLlib\
**Created:** [February 13, 2022, 6:40am UTC](https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038 "2022-02-13T06:40:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fengxiaoxu96](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fengxiaoxu96/32/1901_2.png) [@fengxiaoxu96](https://discuss.ray.io/u/fengxiaoxu96)\
**Post date:** [February 13, 2022, 6:40am UTC](https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038/1 "2022-02-13T06:40:50Z")

</div>

Hi,

I am using RLlib for training my own robot agent.  
I want to ask how to obtain the variance of reward over a period, maybe as same as the period of computation of `episode_reward_mean`.

It will be better If the variance can be shown in tensorboard so that I can postprocess and plot it after training.

Appreciate any idea about this!

---

<div class="post-metadata">

**Author:** ![gjoliver](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gjoliver/32/1490_2.png) [@gjoliver](https://discuss.ray.io/u/gjoliver)\
**Post date:** [February 14, 2022, 8:34pm UTC](https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038/2 "2022-02-14T20:34:48Z")

</div>

> [@fengxiaoxu96](#):
>
> episode\_reward\_mean

This is a good place to start looking, maybe you can contribute a patch for episode\_reward\_stddev to RLlib? 🙂

> <https://github.com/ray-project/ray/blob/master/rllib/evaluation/metrics.py#L229-L231>

---

<div class="post-metadata">

**Author:** ![fengxiaoxu96](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fengxiaoxu96/32/1901_2.png) [@fengxiaoxu96](https://discuss.ray.io/u/fengxiaoxu96)\
**Post date:** [February 15, 2022, 3:34am UTC](https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038/3 "2022-02-15T03:34:47Z")

</div>

Hi, `gjoliver`.

Thanks a lot for your helpful information.

Based on the code you mentioned, I have a very simple idea to add stddev computation lines into following part:

```auto
reward_stddev = np.std(episode_rewards)

```

```auto
reward_stddev = float("nan")

```

> <https://github.com/ray-project/ray/blob/8f9e0d7f6bbe4d0b0610826ad8cd22922397ffc0/rllib/evaluation/metrics.py#L173-L180>

and then the `return` dict of this function should also be added with

```auto
episode_rewards_stddev = reward_stddev,

```

One thing I want to ask is about tensorboard.  
I checked the code in `tune/logger` and `train/callbacks/logging` and  
the data should be added into tensorboard if it is the instance of one of predefined data classes.

But I didn’t see like `policy_reward_min` item in my previous experiments, so why was that?  
Is it because the data corresonding to `policy_reward_min` does not belong to limited data classes, or there is other processing that stops these items being added into tensorboard?

---

<div class="post-metadata">

**Author:** ![gjoliver](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gjoliver/32/1490_2.png) [@gjoliver](https://discuss.ray.io/u/gjoliver)\
**Post date:** [February 15, 2022, 5:32am UTC](https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038/4 "2022-02-15T05:32:01Z")

</div>

I double checked with the team, as long as your metrics are part of the result dict that is returned to Tune, TBXLogger will automatically log it to the tensorboard output file:

> <https://github.com/ray-project/ray/blob/master/python/ray/tune/logger.py#L183>

---

<div class="post-metadata">

**Author:** ![fengxiaoxu96](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fengxiaoxu96/32/1901_2.png) [@fengxiaoxu96](https://discuss.ray.io/u/fengxiaoxu96)\
**Post date:** [February 15, 2022, 6:11am UTC](https://discuss.ray.io/t/add-episode-reward-variance-into-matrix-and-tensorboard/5038/5 "2022-02-15T06:11:24Z")

</div>

Thank you for this reply.  
I will check my own code for that.