# Masking Invalid Actions for DQN Algorithm

**URL:** <https://discuss.ray.io/t/masking-invalid-actions-for-dqn-algorithm/9844>\
**Category:** Ray Tune\
**Created:** [March 22, 2023, 2:27pm UTC](https://discuss.ray.io/t/masking-invalid-actions-for-dqn-algorithm/9844 "2023-03-22T14:27:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Archana\_R](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/archana_r/32/3829_2.png) [@Archana\_R](https://discuss.ray.io/u/Archana_R)\
**Post date:** [March 22, 2023, 2:27pm UTC](https://discuss.ray.io/t/masking-invalid-actions-for-dqn-algorithm/9844/1 "2023-03-22T14:27:36Z")

</div>

I am not able to successfully mask all the invalid actions using DQN Algorithm.  
Please find the custom model created

class ActionMaskModel(DistributionalQTFModel):

```
def __init__ (self, obs_space, action_space, num_outputs,
             model_config, name, true_obs_shape=(870,), 
             action_embed_size=144,**kw): 
    super(ActionMaskModel, self). __init__ (obs_space, action_space, num_outputs, model_config, name, **kw)
    print("action_space :", action_space)
    print("model_config :", model_config)
    self.action_embed_model = FullyConnectedNetwork(
        Box(-1, 1, shape=true_obs_shape),
        action_space,
        action_embed_size,
        model_config,
        name + "_action_embed",
    )

def forward(self, input_dict, state, seq_lens):

    # Extract the available actions tensor from the observation.
    avail_actions = input_dict["obs"]["avail_actions"]
    action_mask = input_dict["obs"]["action_mask"]

    # Compute the predicted action embedding
    action_embed, _ = self.action_embed_model({"obs": input_dict["obs"]["state"]})

    # Expand the model output to [BATCH, 1, EMBED_SIZE]. Note that the
    # avail actions tensor is of shape [BATCH, MAX_ACTIONS, EMBED_SIZE].
    intent_vector = tf.expand_dims(action_embed, 2)

    # Batch dot product => shape of logits is [BATCH, MAX_ACTIONS].
    action_logits = tf.reduce_sum(avail_actions * intent_vector, axis=1)

    # Mask out invalid actions (use tf.float32.min for stability)
    inf_mask = tf.maximum(tf.math.log(action_mask), tf.float32.min)
    return action_logits + inf_mask, state

```

Am i missing anything here ? I am even inheriting the DistributionalQTFModel

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [March 22, 2023, 6:59pm UTC](https://discuss.ray.io/t/masking-invalid-actions-for-dqn-algorithm/9844/2 "2023-03-22T18:59:50Z")

</div>

Hi @Archana_R,

Action making will not work like that for DQN. Those values you are masking are intermediate activations they are not the policy logits for the actions. The type of making you are doing only works for the on policy models like PPO or A2C.

---

<div class="post-metadata">

**Author:** ![Archana\_R](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/archana_r/32/3829_2.png) [@Archana\_R](https://discuss.ray.io/u/Archana_R)\
**Post date:** [March 22, 2023, 7:14pm UTC](https://discuss.ray.io/t/masking-invalid-actions-for-dqn-algorithm/9844/3 "2023-03-22T19:14:31Z")

</div>

How do you suggest I do it for DQN ?