# How does "rollout\_fragment\_length" in the specification for the trainer interact with "max\_seq\_len" in the specification for the model? **URL:** \ **Category:** RLlib\ **Created:** [June 1, 2021, 11:22pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375 "2021-06-01T23:22:40Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![LukasNothhelfer](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lukasnothhelfer/32/1062_2.png) [@LukasNothhelfer](https://discuss.ray.io/u/LukasNothhelfer)\ **Post date:** [June 1, 2021, 11:22pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/1 "2021-06-01T23:22:40Z")
The `max_seq_len` specifies which maximum sequence length the model gets during the training. `rollout_fragment_length` specifies how many time steps an episode is rolled out. I wonder how both configurations interact with each other for the following cases. (1) `max_seq_len` \> `rollout_fragment_length` (2) `max_seq_len` \< `rollout_fragment_length` (3) `max_seq_len` == `rollout_fragment_length` I have already found that the model contains at most `max_seq_len` time steps of the trajectories when `rollout_fragment_length` \> `max_seq_len`. But what happens to the rest of the trajectories? Are the trajectories cut into pieces so that the model always gets at most `max_seq_len` time steps? If so, where is this implemented? ---
**Author:** ![smorad](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/smorad/32/272_2.png) [@smorad](https://discuss.ray.io/u/smorad)\ **Post date:** [June 4, 2021, 10:29am UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/2 "2021-06-04T10:29:14Z")
Afaik, max\_seq\_len chunks the batch into pieces for the backward pass. After each chunk, the state is exported for the next backward pass. So it really only affects memory usage and computational speed. Sven may know better. ---
**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\ **Post date:** [June 4, 2021, 12:34pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/3 "2021-06-04T12:34:41Z")
Hi Lukas, hi smorad, I have a slightly different understanding of the two parameters. The `rollout_fragment_length` describes how many steps a rollout worker has to do before these experiences can be collected. Two examples: - Let’s say a rollout worker has zero experiences collected - it is just starting to act. If it completes an episode of m\>n steps, where n is the `rollout_fragment_length`. It will chop this episode into pieces of length n and they will be collected by the training algorithm. It then leaves the left over experiences to be added to it’s future experiences and repeats the process. - In another scenario, the the `rollout_worker` also starts with zero experiences, but his first episode is shorter than the `rollout_fragment_length`. It must therefore collect more experiences before it can send a single fragment of experiences consisting of experiences from multiple episodes. RLlib keeps track of what experiences belong to what episode for you. So later on we avoid feeding recurrent models experiences from different episodes if we do not want to do that. `max_seq_len` is predominantly used in [ray/rnn\_sequencing.py at master · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/master/rllib/policy/rnn_sequencing.py) to chop sequences of size `max_seq_len`. You need this especially for recurrent models. As far as I know, it can be ignored if your policy is stateless. (1) Since `max_seq_len` is larger than the largest fragment you have collected, RLlib will use the largest sequence length it can find in your batch - possibly `rollout_fragment_length` itself, if your episodes are long enough. Or you happen to have a sequence that stems from two rollouts fragments if you are “lucky” and can use your maximum sequence length. (2) RLlib looks for the largest sequence in the batch and chops the batch into pieces of that size. (3) Again, if your episodes are long enough, your sequences will be of size `rollout_fragment_length`. Otherwise they will be smaller. The trajectories are cut into pieces and the recurrent model (or attention model?) receives sequences of at most `max_seq_len` timesteps. But as far as I understand it, nothing goes to waste. Smaller chunks are just padded if they need to be. @smorad So for my understanding, this affects memory usage. But it might also affect the performance of the model, because of the padding? ---
**Author:** ![smorad](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/smorad/32/272_2.png) [@smorad](https://discuss.ray.io/u/smorad)\ **Post date:** [June 7, 2021, 3:58pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/4 "2021-06-07T15:58:04Z")
You’re right, I didn’t think about the padding but it could certainly affect the performance for certain cases. ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [June 7, 2021, 11:06pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/5 "2021-06-07T23:06:05Z")
By performance do we mean speed or gradient computations? If it is speed then I agree that padding will make is slower. If it gradients that should be fine because the padded loss values are either zeroed out or removed before the final accumulation operation. ---
**Author:** ![vakker00](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vakker00](https://discuss.ray.io/u/vakker00)\ **Post date:** [June 16, 2021, 2:32pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/6 "2021-06-16T14:32:11Z")
One additional thing to add to this is the `train_batch_size`, which is related. The param `train_batch_size` specifies how many timesteps are passed to the training iteration. However, if `rollout_fragment_length` (200 by default) is larger than `train_batch_size` (let’s say 20), then RLlib will still collect that 200 timesteps (but only use 20 for the training?). Probably this is not an issue generally, but things didn’t add up for me when I tried to have some quick debugging runs with low `train_batch_size`. ---
**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\ **Post date:** [July 14, 2021, 2:36pm UTC](https://discuss.ray.io/t/how-does-rollout-fragment-length-in-the-specification-for-the-trainer-interact-with-max-seq-len-in-the-specification-for-the-model/2375/7 "2021-07-14T14:36:41Z")
I would like to add that there is [a good read on this in the documentation](https://docs.ray.io/en/master/rllib-sample-collection.html).