SAC Training Performance Detirioration

RLlib
#1

Hey guys,

Disclaimer, first time working with an RL agent !

For my project we are using an SAC agent to drive a vehicle in a simulation. Long story short, I’ve had 0 success getting my agent to reach an optimal state. Furthermore, the agent seems to be performing great in the beginning and then its performance drops drastically later in training. This run used the default HPs for SAC.

Due to a memory leak with the sim that I;m using I can only run the agent for about about 20K steps before having to save the last checkpoint and restore.

image
image2101×409 32 KB

image
image2158×418 38.2 KB

Thanks

#2

Hi there! :wave:t3:

Would you like to ask your question in RLlib Office Hours? :writing_hand:t3: Just add your question to this doc: RLlib Office Hours - Google Docs

Thanks! Hope to see you there!

1 Like
#3

Hi Sebastian, I moved your question to July 5th Office Hours, hope that’s OK? Let me know if you need help getting unstuck sooner.

Thanks!
Christy

#4

Hey @Stale_neutrino , thanks for posting this question.
If you have an unstable environment, you may want to try some of RLlib’s fault tolerance settings described here:

  • recreate_failed_workers=True
  • restart_failed_sub_environments=True
  • ignore_worker_failures=True

In order for these settings to work, though, you would need some remote workers setup.
For SAC, you usually wouldn’t use many remote workers (default is actually 0, meaning just use the “local” one), but maybe you can try it with at least a few remote workers: num_workers=2 or so.

On the collapsing performance: Not sure what this could be. Hard to tell from this distance.