# Are there some blocking points in adding LSTM to DQN?

**URL:** <https://discuss.ray.io/t/are-there-some-blocking-points-in-adding-lstm-to-dqn/1507>\
**Category:** RLlib\
**Created:** [April 1, 2021, 7:58am UTC](https://discuss.ray.io/t/are-there-some-blocking-points-in-adding-lstm-to-dqn/1507 "2021-04-01T07:58:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maxime\_Riche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/maxime_riche/32/731_2.png) [@Maxime\_Riche](https://discuss.ray.io/u/Maxime_Riche)\
**Post date:** [April 1, 2021, 7:58am UTC](https://discuss.ray.io/t/are-there-some-blocking-points-in-adding-lstm-to-dqn/1507/1 "2021-04-01T07:58:36Z")

</div>

I will probably change the DQN policy & Trainer to support LSTM (or at least RNN).

Do you know major problems preventing that?  
I am asking since DQN doesn’t currently support the RNN and LSTM options in RLLib. I would like to be sure that the is no serious known issue preventing that.

---

<div class="post-metadata">

**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\
**Post date:** [April 1, 2021, 2:56pm UTC](https://discuss.ray.io/t/are-there-some-blocking-points-in-adding-lstm-to-dqn/1507/2 "2021-04-01T14:56:50Z")

</div>

@sven1977 @ericl could you chime in here?

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [April 9, 2021, 3:03pm UTC](https://discuss.ray.io/t/are-there-some-blocking-points-in-adding-lstm-to-dqn/1507/3 "2021-04-09T15:03:22Z")

</div>

Hey @Maxime_Riche , great question! 🙂  
We do have an R2D2 agent (since a few weeks ago) in the current master (and upcoming 1.3 release).  
It’s basically a vanilla DQN that runs with LSTMs/RNNs by storing sequences (of len `max_seq_len`) in the buffer, then samples these sequences as-is from the buffer and runs them through the LSTM for learning updates. The paper we used is linked here:  
[https://openreview.net/pdf?id=r1lyTjAqYX](https://openreview.net/pdf?id=r1lyTjAqYX)

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [November 15, 2021, 7:53pm UTC](https://discuss.ray.io/t/are-there-some-blocking-points-in-adding-lstm-to-dqn/1507/4 "2021-11-15T19:53:23Z")

</div>

@sven1977 ,

as far as I understand it that means then that a `max_seq_len=1` would make the LSTM layer meaningless?