R2D2 with Prioritized Replay Buffer

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

The following is an extract from r2d2.py:

self.replay_buffer_config = {
            "type": "MultiAgentReplayBuffer",
            # Specify prioritized replay by supplying a buffer type that supports
            # prioritization, for example: MultiAgentPrioritizedReplayBuffer.

Note the 2 bottom lines (the comments), which talk about PrioritizedReplayBuffer.

My R2D2 code works well with the default MultiAgentReplayBuffer. However, when I run my code with MultiAgentPrioritizedReplayBuffer, I get:

ValueError: Prioritized replay is not supported when replay_sequence_length > 1.

As far as I’m aware there is no way to use R2D2 with replay_sequence_length <= 1.

Is my understanding correct then that R2D2, and any other algo using an RNN, can’t use Prioritized Replay?

To me it’s quite an unpleasant surprise, because in the Rainbow paper they demonstrate that Prioritized Replay is responsible for the biggest improvement out of all the Rainbow improvements.

2

Just found this example which demonstrates R2D2 with the MultiAgentPrioritizedReplayBuffer: rllib/examples/replay_buffer_api.py

When I copy/paste and run this, I get ValueError: Prioritized replay is not supported when replay_sequence_length > 1.

Ray 2.6.0