# Custom LSTM Model, how to define the SEQ\_LEN **URL:** \ **Category:** RLlib\ **Created:** [April 3, 2022, 2:51pm UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657 "2022-04-03T14:51:11Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ColdFrenzy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/coldfrenzy/32/683_2.png) [@ColdFrenzy](https://discuss.ray.io/u/ColdFrenzy)\ **Post date:** [April 3, 2022, 2:51pm UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657/1 "2022-04-03T14:51:11Z")
**How severe does this issue affect your experience of using Ray?** - Low: It annoys or frustrates me for a moment. Hello everyone, I’m trying to create a custom actor-critic model with LSTM similar to this: > But with the addition of 2 different LSTMs one for the actor and one for the critic as in > [@Best way to have custom value state + LSTM](https://discuss.ray.io/t/best-way-to-have-custom-value-state-lstm/2029/6): > > Hi @mannyv, thanks for your quick response! I have an idea and I am just about to run my custom model to see if this will work. Would it be ok if I just concat my value\_hidden\_states with action\_hidden\_states and send them together away upon each forward\_run call? More concretely: # Two separate LSTMs with two separate branches... self.actor\_layers[-1].\_model[0].weight.new(1, self.lstm\_state\_size).zero\_().squeeze(0), self.actor\_layers[-1].\_model[0].weight.new(1, self… One thing that is not clear to me is how should I set the time length of the LSTM. It looks like rllib by default set it to 32 in my code, but actually in my configuration i never used that value: ```auto "use_lstm": False, "max_seq_len": 20, "lstm_cell_size": 256, "lstm_use_prev_action": False, "lstm_use_prev_reward": False, "_time_major": False, "custom_model_config": { "share_weights": False, "shared_fc_layers": ([128, 64, 32],), "fc_layers": ([], []), "cell_size": 30 }, "rollout_fragment_length": 20 if args.debug else 200, "train_batch_size": 400 if args.debug else 4000, "sgd_minibatch_size": 25 if args.debug else 256, "shuffle_sequences": True, "num_sgd_iter": 30, ``` Do you know how to control such parameter in a custom LSTM model? ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [April 3, 2022, 3:54pm UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657/2 "2022-04-03T15:54:53Z")
Hi @ColdFrenzy, Can you share how you determined the 32 value? The time length will vary depending on if you are in the sample phase or the train phase. In the sample phase the time dimension will be 1 because rllib generates actions for each step on at a time. During the training phase, which your configuration the max size of the time dimension will be 20 based in your max\_seq\_len setting. That also serves as the truncation length for tbptt. It is likely you will have sequences that are shorter than 20 though. This will happen if your episode is shorter than 20 or if you are using the truncate\_batch sample setting and it pauses in the middle of an episode because you have hit your rollout\_fragment\_length. Hope that helps. * * * > **[Manny is helping people do cool things with RL](https://www.buymeacoffee.com/mannyv)** > > Hey, so glad we got to connect. If you found my suggestions useful feel free to buy me a coffee! ---
**Author:** ![ColdFrenzy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/coldfrenzy/32/683_2.png) [@ColdFrenzy](https://discuss.ray.io/u/ColdFrenzy)\ **Post date:** [April 3, 2022, 6:39pm UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657/3 "2022-04-03T18:39:34Z")
Hi @mannyv , thanks for the info. Actually, I inserted a breakpoint inside the forward call of my custom LSTM model and I saw the shape of the seq\_len which is a torch tensor of size 32 with all ones. From what you are telling me, the values 1 are probably the time dimension since I’m in the sampling phase. The strange thing then is that the batch\_size is 32, but I don’t think I have set it to 32 in my config. I tried to continue with debugging and at some point the seq\_len becomes tensor ([8, 8, 8, 8], dtype = torch.int32). P.S. just to give a little bit of context, in my environment i have three agents which observe a state space of dimension 18, 9 are the real observations and 9 are the action\_mask observations. So in the end I’d like to understand a little better how batch sizes and time dimensions are handled in general, not necessarily in the context of my environment. EDIT: By following the function calls, I’ve ended up here > where batch size is defined as 32 and consequently the seq\_len is defined as > ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [April 4, 2022, 12:18am UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657/4 "2022-04-04T00:18:29Z")
@ColdFrenzy, Follow my posts in this thread. They might have some of the info you are interested in. That code is some initialization code There are about 3 calls to forward at the very beginning before training starts that are used to set up ViewRequirements. > [@\[rllib\] SampleBatch "state\_in\_0" dimension shorter than expected](https://discuss.ray.io/t/rllib-samplebatch-state-in-0-dimension-shorter-than-expected/2354/4): > > What is your max\_seq\_len? Before passing the model into forwad\_rnn two things happen. 1. The data in the sample batch is padded so that all inputs are the same size as your longest episode. So if you had sequences of length [5,12,10,4] they would all be padded with zeros to be 12 steps long. You would end up with a total of12 \* 4 timesteps in your sample batch. They are also shortened to be no larger than max sequence length. Let’s say your max\_seq\_len in the model config was 20. Your state in … ---
**Author:** ![Federica\_Tonti](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/federica_tonti/32/6094_2.png) [@Federica\_Tonti](https://discuss.ray.io/u/Federica_Tonti)\ **Post date:** [May 9, 2024, 7:42pm UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657/5 "2024-05-09T19:42:45Z")
Hi @ColdFrenzy i am experiencing the same problem and completely blocked my work, I am getting crazy ☹ can you maybe explain how did you solve the issue? Thanks! ---
**Author:** ![Marko\_Blanusa](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/marko_blanusa/32/6103_2.png) [@Marko\_Blanusa](https://discuss.ray.io/u/Marko_Blanusa)\ **Post date:** [June 10, 2024, 9:34pm UTC](https://discuss.ray.io/t/custom-lstm-model-how-to-define-the-seq-len/5657/6 "2024-06-10T21:34:49Z")
Me too, I’m getting the exact same problem with the batch being 32 by default when arriving in the forward method with the seq\_lens tensor being full of 1s. And I suspect this is causing the error : File “C:\Users\marko\anaconda3\envs\rllib-torch\lib\site-packages\ray\rllib\evaluation\worker\_set.py”, line 181, in **init** raise e.args[0].args[2] RuntimeError: shape ‘[5504, 1]’ is invalid for input of size 12288 I don’t know where this shape and input size are coming from. My environment outputs states of shape (96, 36) representing one sequence per step. My entire dataset is batched with a size of 128. If anyone has an idea that would be a big help.