# SAC Agent 'Forgets' During Training

**URL:** <https://discuss.ray.io/t/sac-agent-forgets-during-training/7539>\
**Category:** RLlib\
**Created:** [September 13, 2022, 3:09pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539 "2022-09-13T15:09:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stale\_neutrino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/stale_neutrino/32/2263_2.png) [@Stale\_neutrino](https://discuss.ray.io/u/Stale_neutrino)\
**Post date:** [September 13, 2022, 3:09pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539/1 "2022-09-13T15:09:58Z")

</div>

Hey guys,

I’m currently training an SAC agent (4D continuous action space [-1,1] , 19 D obs space [unbound]) which is able to create successful episodes fairly quickly but then it starts to perform poorly after awhile (See graph bellow)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/d/dc55d9d93481e5874cd476526691e2536ac1830d.png)

- horizontal axis is the episode number

For this agent I’m using the default SAC config  
Trained for 155005 steps

What is the best course of action to take when you have an agent exhibiting this behavior ?

Ray version: 2.0  
Python: 3.9

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [September 13, 2022, 5:20pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539/2 "2022-09-13T17:20:51Z")

</div>

Hi @Stale_neutrino,

- What parameters have you tuned? What came out of that?
- tau?
- target\_network\_update\_freq?
- LR schedule?

@avnishn can we get some tuning advice from you?

---

<div class="post-metadata">

**Author:** ![avnishn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/avnishn/32/1544_2.png) [@avnishn](https://discuss.ray.io/u/avnishn)\
**Post date:** [September 13, 2022, 6:12pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539/3 "2022-09-13T18:12:04Z")

</div>

I’d need to see more data here other than reward curves, so I’ll give some general advice.

Online RL fails to learn when agents don’t explore the optimal behavior. So with that said, we can look at the hparams that control exploration.

In RLlib they are called:  
log\_alpha\_value  
alpha\_value

and then some other ones that would be nice to look at are mean\_q, target\_entropy, actor\_loss, critic\_loss

Also, can you use tensorboard dev and share your tensorboard logs?

---

<div class="post-metadata">

**Author:** ![Stale\_neutrino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/stale_neutrino/32/2263_2.png) [@Stale\_neutrino](https://discuss.ray.io/u/Stale_neutrino)\
**Post date:** [September 13, 2022, 6:45pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539/4 "2022-09-13T18:45:04Z")

</div>

> [@avnishn](#):
>
> learn when agents don’t explore the optimal behavior. So with that said, we can look at the hparams th

@avnishn [TensorBoard.dev - Upload and Share ML Experiments for Free](https://tensorboard.dev/experiment/nggKEDXJSkGx3GjCNGTgBQ/#scalars)

---

<div class="post-metadata">

**Author:** ![Stale\_neutrino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/stale_neutrino/32/2263_2.png) [@Stale\_neutrino](https://discuss.ray.io/u/Stale_neutrino)\
**Post date:** [September 13, 2022, 6:47pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539/5 "2022-09-13T18:47:02Z")

</div>

Hey @arturn,

Left all params as default, that being said I should probably run a tune session for this agent. Besides the params you listed which other should I tune and what should their ranges ?

Thanks

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [September 13, 2022, 7:09pm UTC](https://discuss.ray.io/t/sac-agent-forgets-during-training/7539/6 "2022-09-13T19:09:12Z")

</div>

Hi @Stale_neutrino ,

Have a look at our [tuned examples section](https://github.com/ray-project/ray/tree/master/rllib/tuned_examples) in the repo to find some examples of what parameters we modified in the past and also to find out a good starting point for a hopefully similar problem.

Cheers