Does rllib QMIX work with a tuple of 2 actions?

RLlib
1

I tried to use an action space tuple of 2 discrete actions

ACTION_SPACE = gym.spaces.Tuple([
    gym.spaces.Discrete(3), 
    gym.spaces.Discrete(3), 
])

then got this error:

ValueError: The two structures don't have the same nested structure.

First structure: type=tuple str=(0,)

Second structure: type=tuple str=(Discrete(3), Discrete(3))

More specifically: The two structures don't have the same number of elements. First structure: type=tuple str=(0,). Second structure: type=tuple str=(Discrete(3), Discrete(3))
Entire first structure:
(.,)
Entire second structure:
(., .)

If I change action space to:

ACTION_SPACE = gym.spaces.Tuple([
    gym.spaces.Discrete(3), 
])

then it runs without errors

Sorry this isn’t an reproducible script, I would like to confirm whether QMIX algorithm work with more than 1 discrete action. If so, is there an example of it? the two_step_game example uses 1 discrete action

2

Hi @Jiffer_PengPeng,

Try using MultiDiscrete instead of a Tuple of Discrete.

3

Hi @mannyv
Thanks so much for the quick reply!
I tried MultiDiscrete

ACTION_SPACE = gym.spaces.Tuple([
    gym.spaces.MultiDiscrete([3, 5])
])

then got this error:

ValueError: QMix requires a discrete action space, got MultiDiscrete([3 5])

Also tried removing the tuple:

ACTION_SPACE = gym.spaces.MultiDiscrete([3, 5])

then got

ValueError: Action space must be a Tuple, got MultiDiscrete([3 5]). Use MultiAgentEnv.with_agent_groups() to group related agents for QMix.

Note I don’t think the “Use MultiAgentEnv.with_agent_groups() to group related agents for QMix.” part is related to the problem as i did grouping. This part of error message isn’t accurate

4

How many agents are in your environment? QMIX is expecting a Tuple space with one entry per agent. I am not 100% positive but looking at this I think it will only accept Discrete action spaces for each agent.

1 Like
5

@mannyv
I have 2 agents, 1 predator and 1 prey
The following is the env creation code. RLlibHiWayEnv is a custom env that inherits ray MultiAgentEnv

    PREDATOR_IDS = ["PRED1"]
    PREY_IDS = ["PREY1"]
    grouping = {
        'PREY1': PREY_IDS,
        'PRED1': PREDATOR_IDS,
    }
    env = RLlibHiWayEnv(config)
    return env.with_agent_groups(
        grouping, obs_space=OBSERVATION_SPACE, act_space=ACTION_SPACE
    )

If QMIX accepts two discrete action spaces for each agent, it would work(both action can be homogeneous as well). This is also what I think QMIX’s doc says it can do. But currently it looks like QMIX only accepts 1 discrete action per agent.

6

This will almost certainly hurt performance but if your action space is small you could define your action space as Discrete with d1*d2 actions then map them to (0,0), (0,1),…(0,m),(1,0),…(n,m)

2 Likes
7

This is a pretty good workaround and probably the only workaround. I will start training this way and see it will work. Thanks!

2 Likes