# How are minibatches spliced **URL:** \ **Category:** RLlib\ **Created:** [November 11, 2021, 5:12pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106 "2021-11-11T17:12:33Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 5:12pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/1 "2021-11-11T17:12:33Z")
Hi all, I am currently using a policy/server setup with about 4 remote workers collecting samples, and a single policy server that does the training. I am using the built-in FCNET with LSTM wrapping + PPO, ` "batch_mode": "complete_episodes",` . My questions is as follows: Given the following hypothetical: If my lstm seq\_len is 16 and my minibatch is 16 and my overall buffer is 256. My episodes are all 64 timesteps (4 workers, so 4 episodes in an overall training cycle): 1. When doing the minibatches, will RRllib automatically grab 16 (in order) timesteps from a SINGLE episode to do an SGD pass on? Or will the minibatch consist of 16 RANDOM timesteps from any combination of the episodes? 2. If it is from a single episode, are the 16 timesteps in order to ensure logical time-sequence when learning or will it be any 16 timesteps from a single episode? 2.5) If the 16 are in order, will the batches of each 16 episodes from the episode be in order as well, or is the batch of 16 random? I.E. 0-15, 16-31 OR 0-15, 32-47, 16-31 etc. Thanks in advance! ---
**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\ **Post date:** [November 11, 2021, 5:52pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/2 "2021-11-11T17:52:39Z")
Hi @Denys_Ashikhin , I had a question where in his [answer](https://discuss.ray.io/t/read-tune-console-output-from-simple-q/3845/5) @mannyv also explained a little the minibatch setup. Maybe his answer helps you to understand better what is going on under the hood. Hope this helps ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 7:20pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/3 "2021-11-11T19:20:46Z")
Hi @Lars_Simon_Zehnder , It was a good start, however, unless I misunderstood (high possibility of that) nothing really pertained to how LSTM’s affect the batching for mini-batches and the exact sequence of them… ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [November 11, 2021, 8:39pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/4 "2021-11-11T20:39:26Z")
Hi @Denys_Ashikhin, What is your rollout\_fragment\_length? 1. Yes you will get contiguous time sequences of length 16 or shorter but with your configuration there should not be any shorter. 2 The minibatches are randomly sampled from the full training\_batch. Since your max\_seq\_len sgd\_minibatch\_size match you will only perform sgd on one sequence per epoch. How many num\_sgd\_iters are you doing? The order of the 16 ts long subsequences will be randomly selected on each sgd iteration. Think about it like this. Take your 256 timesteps that are ordered by time and episode and divide them into 16 groups of length 16. Within each group there will be consecutively ordered timesteps from the same episode. During 1 iteration of training, with your settings you will train with each group as a minibatch. You will do this num\_sgd\_iters times. ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 8:47pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/5 "2021-11-11T20:47:03Z")
That was just a hypothetical situation. With numbers that divide nicely. My rollout doesn’t matter since I have `complete_episodes` so it rolls out episode at a time → this allows for back propogation of the LSTM state across entire episode and not just rollout\_fragments as per: ['rollout\_fragment\_length' and 'truncate\_episodes' · Issue #10179 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/10179#issuecomment-676765542) My actual setup (that I am testing right now): train\_batch\_size: 7680, sgd\_minibatch\_size: 64, num\_sgd\_iter: 10, lstm max\_seq\_len: 16 What I’m hoping is that it takes an in-order (from a single episode) batch (minibatch) of 16 samples, if it is from one episode then all 4 batches (of 16) are in order, or if across different episodes then it still retains order acros iters. However, you mentioned that the batch of 16 could be rather random - how does LSTM then learn order of sequences as a game goes on across an entire episode? P.S. The average episode length is ~700 steps for me ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 8:58pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/6 "2021-11-11T20:58:02Z")
> [@mannyv](#): > > You will do this num\_sgd\_iters times. Also, does this mean in my simple example, with an num\_sgd\_iter=2 I would only use 32 samples before going onto the next epoch (which involves waiting for 256 timesteps - leading to a 224 timesteps wastage???) ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [November 11, 2021, 9:00pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/7 "2021-11-11T21:00:02Z")
No that is not what I meant. You would go through each minibatche twice. ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 9:06pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/8 "2021-11-11T21:06:01Z")
This is the code I have (copied from slightly older rllib docs): ```auto # Number of timesteps collected for each SGD round. This defines the size # of each SGD epoch. "train_batch_size": 7680, # Total SGD batch size across all devices for SGD. This defines the # minibatch size within each epoch. "sgd_minibatch_size": 64, # Number of SGD iterations in each outer loop (i.e., number of epochs to # execute per train batch). "num_sgd_iter": 10, # Whether to shuffle sequences in the batch when training (recommended). "shuffle_sequences": False, ``` I had read that as num\_sgd\_iter: 10 meaning I will go over single datapoint 10 times (so 7680/64 = 120 inner iterations). Then 10 outer ones → this is not correct then? Instead it is 10 \* 64 in MY case, so I would only use 640 out of 7680 samples?? and the rest would be discarded…? ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [November 11, 2021, 9:14pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/10 "2021-11-11T21:14:50Z")
@Denys_Ashikhin, No I was wrong about that your original understanding is correct. Sorry about that. I update the wording of my previous post to be more accurate. ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 9:20pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/11 "2021-11-11T21:20:30Z")
Okay good! Phew lol. And also, is sequence shuffling = false going to mean that batches of 16 seq will be in order for each episode? Is there a reason why we don’t want lstm seq in order for the episode? ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [November 11, 2021, 9:23pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/12 "2021-11-11T21:23:50Z")
The code is here if you want to look at it. > For boundaries past the max\_seq\_len the order does not matter. The lstm will only for a memory over that many timesteps so it does not really matter across the max\_seq\_len if they are in order or not. It looks like in the code that if you are using an lstm it will shuffle the order of groups in the mini-batch and make sure that the data and the starting state are shuffled correctly together ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 9:26pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/13 "2021-11-11T21:26:06Z")
I realise this is getting side tracked, but then my question is-\> the LSTM cell\_state does not get carried through the entire episode? Is there a way to make it go all the way? ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [November 11, 2021, 9:28pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/14 "2021-11-11T21:28:39Z")
You can set it to be longer than your largest possible episode length. But if that length is too long then you have to worry about vanishing gradients in bptt. ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 9:38pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/15 "2021-11-11T21:38:48Z")
Got it, so to summarise: Minibatches will have sequences in order, but the ordering of minibatches will be random. LSTM cell\_state does not get carried across minibatches during training, instead it is limited to max\_seq\_len (after which internal state is reset?) ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [November 11, 2021, 9:45pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/16 "2021-11-11T21:45:45Z")
That is correct. One detail. When we say reset we do not mean to the initial state. It will be whatever the state was during rollouts. It does not loose that knowledge of the state. What is reset (truncated as a more accurate term) is the gradients flowing from previous timesteps. ---
**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\ **Post date:** [November 11, 2021, 9:49pm UTC](https://discuss.ray.io/t/how-are-minibatches-spliced/4106/17 "2021-11-11T21:49:05Z")
Riiight, that makes sense, since for training the inner cell\_state is based on what it was during the moment it was sampled - the training epoch does not modify that value - hence order doesn’t matter too much beyond those gradients. Okay its all clicking together. Last thing (promise) With gpu enabled computation, minibatches are stored in gpu but is the overal batch\_size stored in RAM for GPU (I also also have LZ4 compression enabled)