Potential bug in client server setup with policy mapping functions

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

It set up a client server training setup that follows the cartpole client/server example. The environment is a multi agent setup and therefore uses a policy mapping function. When i run the client and the server setup on the same machine everthing works perfectly for remote and local action inference.

Problem:
When i run the client on another machine it works in remote inference mode but not in local mode. I get:
KeyError: “policy_mapping_fn returned invalid policy id ‘None’!”

My policy function looks like this:

def policy_mapping_fn(agent_id, episode, worker, **kwargs):
    if agent_id.startswith("p_d_"):
        return "pot_decider_policy"
    elif agent_id.startswith("p_m_"):
        return "pot_move_policy"

Theoretically it is possible that it returns None (if the agent_id isn’t matching the patterns in my if statements) but when i add an additional else statement to my mapping function to catch the agent_id that caused the problem i get an agent_id that should actually match my if-statements…

Debugging attempt:

def policy_mapping_fn(agent_id, episode, worker, **kwargs):
    if agent_id.startswith("p_d_"):
        return "pot_decider_policy"
    elif agent_id.startswith("p_m_"):
        return "pot_move_policy"
    else:
        print(agent_id)

Result:
The else statement prints “p_m_0_0” before the error is thrown.

I have no idea what could cause the issue here so if anyone could provide help or a hint where to look it would help me out a lot! Thanks in advance!

HI @Blubberblub,

Is this still an issue or did you figure it out?

Do you have a reproduction script or error stack trace you could share?

@mannyv I didn’t finde the issue so far. So i settled with server-side action generation for now. I didn’t yet find the time to build a reproduction script using the server client example and replacing the env with a multi agent env that actually has a policy_mapping_fn. If the problem occurs as well in this case i will create an issue on github.

Problem was solved by upgrading to ray 2.0.