It seems likely that RLlib’s reproducibility is affected by the type or ordering of agent IDs. There is no documented requirement that agent IDs must be integers—RLlib supports any hashable type, including strings (MultiAgentEnv docs). However, some internal RLlib logic or data structures may iterate over agent IDs in a non-deterministic order if they are strings, especially if sets or dicts are used, which can break reproducibility (discuss.ray.io).
My understanding is that using integer agent IDs may result in a consistent order, while string IDs could lead to unpredictable ordering, affecting random number generation and thus reproducibility. This could be considered a bug or at least an undocumented behavior in RLlib.
Would you like more detail on how to work around this or how to report it upstream?
Sources:
Hint: Mention @RunLLM in the post for followups.