Hi, right now I’m trying to implement an external env using External Environments and RLLink protocol along with Unreal Engine, I did manage to connect and train an example, but came to a question, how am I supposed to implement multiagent envs? Given that the engine is the one responsible to do the stepping, and receives the weights, am I supposed to create a connection and a “server” for every policy? I assume in the engine I have to collect the obs, rewards, actions, etc from all agents that uses the same policy and send it to the server, is that right? What about more than one policy?
Thanks in advance.
Hello @Gonzalo_Diaz ! Welcome to the Ray community~
Have you taken a look at our multi agent environments documentation here? Multi-Agent Environments — Ray 2.47.1
We also have some multi agent examples in the docs: Examples — Ray 2.47.1
If these docs don’t answer your questions I’d be happy to help out further 
Hi Christina. Yes, I have read the examples and the docs, however my question is more related to how can I do a similar functionality than this class: ray/rllib/env/tcp_client_inference_env_runner.py at master · ray-project/ray · GitHub but using Multiagents
I made my env in Unreal Engine work with this tcp env, but I couldn´t find the same implementation for Multiagents. My question more specifically is, do I need to create my own implementation of this class, but using Multiagent Env? or is there already an implementation that I am not aware of?
Thanks
So I don’t think you don’t need to create a separate connection or “server” for each policy. The key is to manage all your agents in a single environment, and send their data to RLlib in a specific structure.
If you have more than 1 policy, you can do it like they described here:
Multi-agent setup: N agents live in the environment and take actions computed by M policy networks. The mapping from agent to policy is flexible and determined by a user-provided mapping function. Here, agent_1 and agent_3 both map to policy_1, whereas agent_2 maps to policy_2.
As far as I can tell with tcp_client_inference_env_runner.py, you’re right that it’s set up for single-agent use. You can try to base your code off the single-agent TCP runner and upgrade it to multi-agent by following the pattern in ExternalMultiAgentEnv, and maybe the Unity3D multi-agent client/server example might be helpful too (pasted below).
Unity3D Multi Agent Example:
server: ray/rllib/examples/envs/external_envs/unity3d_server.py at master · ray-project/ray · GitHub
client: ray/rllib/examples/envs/external_envs/unity3d_client.py at master · ray-project/ray · GitHub
I managed to make it work implementing a modified version of tcp_client_inference_env_runner using MultiAgentEnvRunner, as for ExternalMultiAgentEnv, I decided to not use it because it is marked with @OldApiStack and might be inconvenient in the future.
1 Like