# R2D2 with Prioritized Replay Buffer

**URL:** <https://discuss.ray.io/t/r2d2-with-prioritized-replay-buffer/11520>\
**Category:** RLlib\
**Created:** [July 21, 2023, 7:00pm UTC](https://discuss.ray.io/t/r2d2-with-prioritized-replay-buffer/11520 "2023-07-21T19:00:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ihopethiswillfi](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@ihopethiswillfi](https://discuss.ray.io/u/ihopethiswillfi)\
**Post date:** [July 21, 2023, 7:00pm UTC](https://discuss.ray.io/t/r2d2-with-prioritized-replay-buffer/11520/1 "2023-07-21T19:00:50Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

The following is an extract from `r2d2.py`:

```auto
self.replay_buffer_config = {
            "type": "MultiAgentReplayBuffer",
            # Specify prioritized replay by supplying a buffer type that supports
            # prioritization, for example: MultiAgentPrioritizedReplayBuffer.

```

Note the 2 bottom lines (the comments), which talk about PrioritizedReplayBuffer.

My R2D2 code works well with the default `MultiAgentReplayBuffer`. However, when I run my code with `MultiAgentPrioritizedReplayBuffer`, I get:

`ValueError: Prioritized replay is not supported when replay_sequence_length > 1`.

As far as I’m aware there is no way to use R2D2 with replay\_sequence\_length \<= 1.

Is my understanding correct then that R2D2, and any other algo using an RNN, can’t use Prioritized Replay?

To me it’s quite an unpleasant surprise, because in the Rainbow paper they demonstrate that Prioritized Replay is responsible for the biggest improvement out of all the Rainbow improvements.

---

<div class="post-metadata">

**Author:** ![ihopethiswillfi](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@ihopethiswillfi](https://discuss.ray.io/u/ihopethiswillfi)\
**Post date:** [July 21, 2023, 8:40pm UTC](https://discuss.ray.io/t/r2d2-with-prioritized-replay-buffer/11520/2 "2023-07-21T20:40:49Z")

</div>

Just found this example which demonstrates R2D2 with the `MultiAgentPrioritizedReplayBuffer`: [rllib/examples/replay\_buffer\_api.py](https://github.com/ray-project/ray/blob/master/rllib/examples/replay_buffer_api.py)

When I copy/paste and run this, I get `ValueError: Prioritized replay is not supported when replay_sequence_length > 1.`

Ray 2.6.0