Handling Configurable Multi-Agent vs. Single-Agent Environments

[lukasgppl]

Hey all,

I’m working on a PettingZoo environment that can be configured for either:

• A single central agent, or
• Multiple decentralized agents

I’d like to maintain this flexibility but am unsure about the best approach for RLlib (specifically PPO). Here are the options I’m considering:

1. Use .multi_agent in the PPO config with a single policy for the single agent case – Is this supported in RLlib?
2. Wrap PettingZoo into a Gymnasium env for the single agent case and use the standard PPO single agent config – More work, but keeps configurability.
3. Write a separate Gymnasium env for single-agent – Less elegant but straightforward.

I’d prefer Option 1 if possible—has anyone tried this or know if it’s viable? Are there better alternatives?

Thanks for any insights!

[christina]

Hi lukasgppl! Welcome to the Ray community
Yes what you described in option 1 is viable and I think the best way to go about this! This is supported and works well in RLlib. You can configure your multi_agent settings to have a single policy and map your single agent to that policy. You also maintain one primary environment and RLlib configuration structure whcih will make it easier to update in the future.

So in this case, I would keep one PettingZoo environment and one RLlib multi-agent PPO config to keep things simple.

I think these docs will help too:

• Environments — Ray 2.46.0
  • (specifically the parts about MultiAgentEnv API and config.multi_agent() settings: policies, policy_mapping_fn, policies_to_train)
• Multi-Agent Environments — Ray 2.46.0
  • they talk a bit about the pettingzoo API here which might be helpful for your use case!
• Examples — Ray 2.46.0
  • There’s a few petting zoo examples in here too.

let me know if you still have any trouble or have problems with it!