# Example for action\_masking\_rl\_module broken?

**URL:** <https://discuss.ray.io/t/example-for-action-masking-rl-module-broken/20078>\
**Category:** RLlib\
**Created:** [October 29, 2024, 1:32am UTC](https://discuss.ray.io/t/example-for-action-masking-rl-module-broken/20078 "2024-10-29T01:32:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pitcherrr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/pitcherrr/32/7211_2.png) [@Pitcherrr](https://discuss.ray.io/u/Pitcherrr)\
**Post date:** [October 29, 2024, 1:32am UTC](https://discuss.ray.io/t/example-for-action-masking-rl-module-broken/20078/1 "2024-10-29T01:32:15Z")

</div>

**How severe does this issue affect your experience of using Ray?**  
High: It blocks me to complete my task.

I have recently updated my ray (2.38.0) to migrate towards using the new api. I have been using the action making example to train a PPO agent in a custom environment. When migrating I was getting errors and found that the example for action masking does not run.

To reproduce the error:  
From ray/rllib/examples/rl\_modules  
`python action_masking_rl_module.py --enable-new-api-stack --num-env-runners 2`

The error I am getting is:

```auto
(SingleAgentEnvRunner pid=84919) module = self.module_class( [repeated 2x across cluster]
(SingleAgentEnvRunner pid=84919) TypeError: ActionMaskingRLModule. __init__ () got an unexpected keyword argument 'observation_space' [repeated 2x across cluster]

```

---

<div class="post-metadata">

**Author:** ![Pitcherrr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/pitcherrr/32/7211_2.png) [@Pitcherrr](https://discuss.ray.io/u/Pitcherrr)\
**Post date:** [October 29, 2024, 4:05am UTC](https://discuss.ray.io/t/example-for-action-masking-rl-module-broken/20078/2 "2024-10-29T04:05:03Z")

</div>

It seems that in rl\_module.py TypeError is not handled for model\_config deprecation. However after adding this:  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/5/508c1c8ec381d57aa02de4b0fdbc9a467638c681.png)

the example still errors:

```auto
  File "<...>/ray/rllib/examples/rl_modules/classes/action_masking_rlm.py", line 131, in _preprocess_batch
    action_mask = batch[Columns.OBS].pop("action_mask")
AttributeError: 'Tensor' object has no attribute 'pop'

```

I checked the return types and the environment returns OrderedDict’s as its Obs correctly but returns a tensor at some point causing it to crash.

Some notable warnings that I saw:  
From Gymnasium:

```auto
<...>/gymnasium/envs/registration.py:693: UserWarning: WARN: Overriding environment rllib-single-agent-env-v0 already in registry.

```

```auto
<...>/gymnasium/utils/passive_env_checker.py:275: UserWarning: WARN: The reward returned by `step()` must be a float, int, np.integer or np.floating, actual type: <class 'numpy.ndarray'>

```

And obviously the ones from RLLib:

```auto
WARNING deprecation.py:50 -- DeprecationWarning: `RLModule(config=[RLModuleConfig object])` has been deprecated. Use `RLModule(observation_space=.., action_space=.., inference_only=.., model_config=.., catalog_class=..)` instead. This will raise an error in the future!
WARNING deprecation.py:50 -- DeprecationWarning: `RLModule(config=[RLModuleConfig])` has been deprecated. Use `RLModule(observation_space=.., action_space=.., inference_only=.., learner_only=.., model_config=..)` instead. This will raise an error in the future!

```

---

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wpm/32/4241_2.png) [@wpm](https://discuss.ray.io/u/wpm)\
**Post date:** [March 2, 2025, 5:29pm UTC](https://discuss.ray.io/t/example-for-action-masking-rl-module-broken/20078/3 "2025-03-02T17:29:46Z")

</div>

I had a lot of trouble understanding `action_masking_rl.py` and wrote my own simpler example custom action masking environment in the [Ray Masked project](https://gitlab.com/wpm/ray-masked).