# Multiagent Remove\_Policies synchronisation

**URL:** <https://discuss.ray.io/t/multiagent-remove-policies-synchronisation/12409>\
**Category:** RLlib\
**Created:** [October 11, 2023, 10:55am UTC](https://discuss.ray.io/t/multiagent-remove-policies-synchronisation/12409 "2023-10-11T10:55:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SkgTrip](https://avatars.discourse-cdn.com/v4/letter/s/ad7895/32.png) [@SkgTrip](https://discuss.ray.io/u/SkgTrip)\
**Post date:** [October 11, 2023, 10:55am UTC](https://discuss.ray.io/t/multiagent-remove-policies-synchronisation/12409/1 "2023-10-11T10:55:06Z")

</div>

Hi all,

I’m attempting to construct a multiagent RL trainer that has a learnable policy that plays against a range of saved historic variants of the trained agents, selected from a fixed size menagerie.

At the moment this is achieved through a callback function, for which the on\_train\_result function

1. Adds to a list of stored policy indices of fixed size (so popping off the first element if the list exceeds the memory)
2. Defines a new policy\_mapping\_fn which selects a policy from the list of stored policy indices.
3. If an element has been popped from the list, calling algorithm.remove\_policy(policy\_id=old\_policy\_id) to truncate the list
4. Creating a new policy, getting the state from the current active learning policy, and assigning that to the new policy.
5. Calling algorithm.sync\_weights()

However, step 3 is not working as I would expect. If I remove policies from the policy set, I receive an error stating that the policy\_mapping\_fn has returned an invalid policy\_id. Removing step 3), and just letting the size of the dictionary of the weights associated with the previously identified solutions grow without any pruning works just fine, but obviously this is unlikely to be optimal.

To me this would suggest that I don’t understand how the callback function is working in the context of the distributed solver, and that something has become un-synchronised during the process. I haven’t been able to track down any example of multiagent environments using remove\_policy, so I’m struggling to work out the correct RLLIB way to manage this synchronisation issue, and was wondering if anyone had any experience on this front?

Thanks in advance

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

---

<div class="post-metadata">

**Author:** ![unleex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/unleex/32/8495_2.png) [@unleex](https://discuss.ray.io/u/unleex)\
**Post date:** [June 2, 2026, 11:37am UTC](https://discuss.ray.io/t/multiagent-remove-policies-synchronisation/12409/2 "2026-06-02T11:37:01Z")

</div>

hello, have you found a workaround for this?🙂

---

<div class="post-metadata">

**Author:** ![SkgTrip](https://avatars.discourse-cdn.com/v4/letter/s/ad7895/32.png) [@SkgTrip](https://discuss.ray.io/u/SkgTrip)\
**Post date:** [June 2, 2026, 11:50am UTC](https://discuss.ray.io/t/multiagent-remove-policies-synchronisation/12409/3 "2026-06-02T11:50:45Z")

</div>

Unfortunately the answer is not a great one for Ray - I moved over to Tianshou. I found this much easier to implement in Tianshou than Ray.

Good luck!