# PPO Masking with MultiAgentEnv

**URL:** <https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157>\
**Category:** Uncategorized\
**Created:** [March 27, 2025, 2:46pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157 "2025-03-27T14:46:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![gkaganov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gkaganov/32/7819_2.png) [@gkaganov](https://discuss.ray.io/u/gkaganov)\
**Post date:** [March 27, 2025, 2:46pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157/1 "2025-03-27T14:46:09Z")

</div>

How to use action masking with MultiAgentEnv? I tried modeling it after this example: [ray/rllib/examples/envs/classes/action\_mask\_env.py at master · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/master/rllib/examples/envs/classes/action_mask_env.py). Unfortunately is does not handle multiple agents. I tried this but it does not work:

```auto
self.observation_spaces = {
            "player1": gym.spaces.Dict({
                "board": gym.spaces.Box(low=-1.0, high=1.0, shape=(9,), dtype=np.float32),
                "action_mask": gym.spaces.MultiBinary(9),
            }),
            "player2": gym.spaces.Dict({
                "board": gym.spaces.Box(low=-1.0, high=1.0, shape=(9,), dtype=np.float32),
                "action_mask": gym.spaces.MultiBinary(9),
            }),
        }

```

I am setting up a TicTacToe example so the action space is an array of 9.

---

<div class="post-metadata">

**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\
**Post date:** [March 27, 2025, 7:14pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157/2 "2025-03-27T19:14:15Z")

</div>

Hello and welcome to the Ray community!

How is your action masking set up right now? Each agent should have an observation space that includes both the board state and the action mask. The action should return data on whether or not a move is valid or not, aka, should be updated based on the current board state, indicating which positions are available for a move.

Here is more documentation on the tic-tac-toe example which might help: [Multi-Agent Environments — Ray 2.44.1](https://docs.ray.io/en/latest/rllib/multi-agent-envs.html#example-turn-based-environments)

---

<div class="post-metadata">

**Author:** ![gkaganov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gkaganov/32/7819_2.png) [@gkaganov](https://discuss.ray.io/u/gkaganov)\
**Post date:** [March 27, 2025, 7:35pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157/3 "2025-03-27T19:35:10Z")

</div>

Thank you for the quick response!

Currently I penalize illegal actions with a reward of -5 just like in the example you linked. What I would like to do instead is to add action masking to my code. What would the correct syntax be? The code I provided in my question seems to fit your description but it seems that the structure is not quite right. Is there an example of a MultiAgentEnv with action masking?

---

<div class="post-metadata">

**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\
**Post date:** [March 27, 2025, 8:44pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157/4 "2025-03-27T20:44:48Z")

</div>

What error are you getting specifically, is there a code snippet you can send where I can reproduce the error? Thank you!

---

<div class="post-metadata">

**Author:** ![gkaganov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gkaganov/32/7819_2.png) [@gkaganov](https://discuss.ray.io/u/gkaganov)\
**Post date:** [March 28, 2025, 12:55pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157/6 "2025-03-28T12:55:50Z")

</div>

> <https://github.com/gkaganov/rllib-tictactoe/blob/a94ae926ec72fd2d4330318a83b4029b69a377c4/TicTacToe.py>

---

<div class="post-metadata">

**Author:** ![gkaganov](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gkaganov/32/7819_2.png) [@gkaganov](https://discuss.ray.io/u/gkaganov)\
**Post date:** [March 28, 2025, 1:00pm UTC](https://discuss.ray.io/t/ppo-masking-with-multiagentenv/22157/7 "2025-03-28T13:00:25Z")

</div>

> <https://gist.github.com/gkaganov/183943ebd9956c6e96cc20c77051d229>