Using Hindsight Experience Replay in SAC

RLlib
1

I am training my own agent with SAC.

But I also want to test HER with SAC performance in my case,
so I wonder is there existing API or method to use HER in ray?

If no, how can I insert my customized HER into algorithms?

Appreciate any helps and ideas

2

@arturn could you respond to this, as you’re in the process of implementing the replay buffer api?

3

Hi @fengxiaoxu96,

so I wonder is there existing API or method to use HER in ray?

To the best of my knowledge, RLlib offers no functionality that does what HER does.
@avnishn might know better, since he has been around longer.

If no, how can I insert my customized HER into algorithms?

HER is not an algorithm like for example DQN or DDPG are and therefore should, imho, not be part of what you can find under RLlib’s algorithms section if that is what you where thinking of.

Rather, as @avnishn suggested, this should live in the new Replay Buffer API.
HER is not yet in the scope of this API - would you be willing to contribute?
Otherwise you will have to wait some time until it makes it into RLlib I think.

4

Hi, @arturn
A huge thank you that you can offer this information,
and sorry I missed this message for a long time until now.

I think I am willing to contribute to this if I have the chance,
but I know poorly in how Replay Buffer API works in the whole framework.
So I think I need to look through source code more in details about this part.
If you can provide some introductions that will be best.

Thank you again for your help!

1 Like
5

Hi, no problem. Have a look at ray/rllib/utils/replay_buffers at master · ray-project/ray · GitHub (edited). That’s where the new API lives. It’s not yet in a release build, so you will have to work with nightly build here and install them manually :slight_smile: Let us know if you make progress on this!

6

HI,

thank you for reply.

But I can not access to the above link that it showed ‘404 This is not the web page you are looking for’.

7

Mhhh. The link works for me. If I click on it and even if I copy it manually.

Edit: If it’s still not working you can just go to the repo and follow the path of the URL, i.e. (…)/rllib/utils/replay_buffers

8

https://github.com/anyscale is OK for me,
but https://github.com/anyscale/ray returns 404 error.

9

Sorry, my bad. Here’s the link through the open source git ray/rllib/utils/replay_buffers at master · ray-project/ray · GitHub