# \[RLlib\] Impossible actions **URL:** \ **Category:** RLlib\ **Created:** [February 15, 2021, 3:35pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890 "2021-02-15T15:35:24Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Jogima-cyber](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jogima-cyber/32/454_2.png) [@Jogima-cyber](https://discuss.ray.io/u/Jogima-cyber)\ **Post date:** [February 15, 2021, 3:35pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/1 "2021-02-15T15:35:24Z")
Hi there, I’d like to set in a multienv impossible actions. I’ve read the doc ([https://docs.ray.io/en/master/rllib-models.html#variable-length-parametric-action-spaces](https://docs.ray.io/en/master/rllib-models.html#variable-length-parametric-action-spaces)) but I don’t understand the purposes of avail\_actions and action\_mask. Could someone explain it to me please ? ---
**Author:** ![jahabrewer](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jahabrewer/32/504_2.png) [@jahabrewer](https://discuss.ray.io/u/jahabrewer)\ **Post date:** [February 21, 2021, 8:08pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/2 "2021-02-21T20:08:47Z")
I’m in a similar situation. Disclaimer: I know very little about RL, this is just what I’ve pieced together over a few hours googling. `avail_actions` seems to be there for action embeddings. If you follow links in the docs enough, you’ll get to [`ParametricActionsCartPole`](https://github.com/ray-project/ray/blob/d8c32be449653ba38624fb4bba96f34a4365ca0b/rllib/examples/env/parametric_actions_cartpole.py#L7). `action_mask` is what we really want. Unfortunately, this example interweaves it with action _embedding_. I would imagine you could delete `self.action_assignments` and its friends to get to base, mask-only functionality. You’d also need to modify `ParametricActionsModel`, since it expects `avail_actions` in observations and uses it to compute `intent_vector`, and thus `action_logits`. The theory here seems simple–to mask, just intercept `forward` calls and make the logits for masked/invalid actions very negative. I’m not sure why I can’t crack it. Probably a silly dimensions issue. There’s a good [blog post](https://towardsdatascience.com/action-masking-with-rllib-5e4bec5e7505) on this, but it only has one line on `avail_actions`: > The available actions correspond to each of the five items the agent can select for packing. The author seems to work around `avail_actions` rather than excising it; they always set it to ones. Maybe that’s the easier approach. If any maintainers read this, I’d love to see an example with action masking and embedding separated. I’m sure it’s painfully obvious to experts how to separate them. ---
**Author:** ![Jogima-cyber](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jogima-cyber/32/454_2.png) [@Jogima-cyber](https://discuss.ray.io/u/Jogima-cyber)\ **Post date:** [February 21, 2021, 8:53pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/3 "2021-02-21T20:53:08Z")
Thank you very much for your answer, I’ve read through the article, but unfortunately it doesn’t explain the tricky parts at all. I still have no idea what action embedding is. I manage to mask out impossible actions by using action\_mask like that : ``` inf_mask = torch.clamp(torch.log(action_mask), FLOAT_MIN, FLOAT_MAX) return output+inf_mask, [] ``` (it’s in an actor-critic network, output are the logits behind the policy). But I wonder if I’m not missing something important to make everything work with avail\_actions and actions embedding. ---
**Author:** ![jahabrewer](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jahabrewer/32/504_2.png) [@jahabrewer](https://discuss.ray.io/u/jahabrewer)\ **Post date:** [February 22, 2021, 5:08pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/4 "2021-02-22T17:08:17Z")
Yeah, I sympathize. I still don’t quite grok, but I did find this post a bit enlightening: [https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five/](https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five/) ---
**Author:** ![Jogima-cyber](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jogima-cyber/32/454_2.png) [@Jogima-cyber](https://discuss.ray.io/u/Jogima-cyber)\ **Post date:** [February 26, 2021, 8:58am UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/5 "2021-02-26T08:58:19Z")
I love your article!! It’s been a long time since I wanted to have an application of attention mechanisms to reinforcement learning. And also to have an application of reinforcement learning to a complex and variable space of observation as well as to a complex and variable space of action. ---
**Author:** ![klausk55](https://avatars.discourse-cdn.com/v4/letter/k/f17d59/32.png) [@klausk55](https://discuss.ray.io/u/klausk55)\ **Post date:** [April 20, 2021, 1:34pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/6 "2021-04-20T13:34:11Z")
This “available actions thing” seems to be helpful when you have to handle huge action spaces and/or a varying number of available actions during steps. A small example how I interpret it: all\_actions = {0, 1, 2, 3, 4, 5}, n=6 total number of actions, action\_embedding\_size=2 =\> action embedding matrix E is 6x2 m=3 \< n available action at a specific timestep, e.g. avail\_actions=(0, 2, 4) =\> action embedding for avail\_actions is a matrix E\* composed of 1st, 3rd and 5th row of E Finally, you calculate the dot product of the intent vector from the NN (with action\_embedding\_size) and E\* Code example showing the embeddings: ``` import numpy import tensorflow as tf model = tf.keras.Sequential() embed = tf.keras.layers.Embedding(6, 2) model.add(embed) model.summary() print(embed.get_weights()) input = numpy.asarray_chkfinite([0, 2, 4]) model.compile("rmsprop", "mse") out = model.predict(input) for i in range(out.shape[0]): print("{} {}".format(input[i], out[i])) Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (None, None, 2) 12 ================================================================= Total params: 12 Trainable params: 12 Non-trainable params: 0 _________________________________________________________________ [array([[-0.04093417, -0.02362244], [-0.01528452, -0.02044444], [0.04733466, 0.01246139], [0.01975517, 0.02948004], [0.03812562, 0.0137356], [0.04121368, -0.01421856]], dtype=float32)] 2021-04-20 15:25:32.082275: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 0 [[-0.04093417 -0.02362244]] 2 [[0.04733466 0.01246139]] 4 [[0.03812562 0.0137356]] ``` ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 20, 2021, 3:02pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/7 "2021-04-20T15:02:49Z")
I authored a paper that has heavy use of invalid action making in complex action spaces. All the examples are using Griddly and RLLib. Paper: [[2104.07294] Generalising Discrete Action Spaces with Conditional Action Trees](https://arxiv.org/abs/2104.07294) RLLib Code: [GitHub - Bam4d/conditional-action-trees: Example Code for the Conditional Action Trees Paper](https://github.com/Bam4d/conditional-action-trees) Might shed some light on action masking and why its required and how you can apply it. ---
**Author:** ![vlainic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vlainic/32/2330_2.png) [@vlainic](https://discuss.ray.io/u/vlainic)\ **Post date:** [March 28, 2022, 6:26pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/8 "2022-03-28T18:26:00Z")
> [@jahabrewer](#): > > If any maintainers read this, I’d love to see an example with action masking and embedding separated. I’m sure it’s painfully obvious to experts how to separate them. I totally found my problem in this statement, as I do not see the purpose of `self.action_embed_model` in [PA model](https://github.com/ray-project/ray/blob/master/rllib/examples/models/parametric_actions_model.py) if one only wants to mask. But I am not managing to make [a mock-up code](https://github.com/ray-project/ray/blob/master/rllib/examples/models/parametric_actions_model.py#L68) run w/o the `self.action_embed_model`. Did you make it at the end @jahabrewer ? ---
**Author:** ![vlainic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vlainic/32/2330_2.png) [@vlainic](https://discuss.ray.io/u/vlainic)\ **Post date:** [March 29, 2022, 7:27pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/9 "2022-03-29T19:27:24Z")
Okay, I am blind… I somehow skipped with my eyes over the first file in the repo folders: > ---
**Author:** ![James\_Liu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/james_liu/32/1415_2.png) [@James\_Liu](https://discuss.ray.io/u/James_Liu)\ **Post date:** [March 30, 2022, 7:28pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/10 "2022-03-30T19:28:12Z")
Thanks for the example. I am wondering if there is any way of action masking for continuous action spaces? ---
**Author:** ![sirjay](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sirjay/32/2486_2.png) [@sirjay](https://discuss.ray.io/u/sirjay)\ **Post date:** [April 6, 2022, 8:18pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/11 "2022-04-06T20:18:33Z")
Who knows what’s the difference between` class ParametricActionsModel(DistributionalQTFModel)` and `class ActionMaskModel(TFModelV2)`? I guess both performs masking invalid actions. > > ---
**Author:** ![vlainic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vlainic/32/2330_2.png) [@vlainic](https://discuss.ray.io/u/vlainic)\ **Post date:** [May 11, 2022, 12:16pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/12 "2022-05-11T12:16:08Z")
In theory, one would use [Parametric Actions with embeddings](https://docs.ray.io/en/latest/rllib/rllib-models.html#variable-length-parametric-action-spaces) for continuous action space. However, I am not sure that RLlib has this implemented as only `Discrete Actions` column in [algo list](https://docs.ray.io/en/latest/rllib/rllib-algorithms.html) has **+parametric** 🙂 ---
**Author:** ![vlainic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vlainic/32/2330_2.png) [@vlainic](https://discuss.ray.io/u/vlainic)\ **Post date:** [May 11, 2022, 12:19pm UTC](https://discuss.ray.io/t/rllib-impossible-actions/890/13 "2022-05-11T12:19:28Z")
`ActionMaskModel` is a simpler version of `ParametricActionsModel`: - The first one just masks some discrete actions following the environmental rules - The second one is more general and can be used on huge discrete action spaces with embeddings or even continuous action space (in theory, not sure does RLlib supports it)