# VTrace: Understanding shape of batches

**URL:** https://discuss.ray.io/t/vtrace-understanding-shape-of-batches/4028
**Category:** RLlib
**Created:** [November 3, 2021, 3:33pm UTC](https://discuss.ray.io/t/vtrace-understanding-shape-of-batches/4028 "2021-11-03T15:33:11Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![LecJackS](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lecjacks/32/28_2.png) [@LecJackS](https://discuss.ray.io/u/LecJackS)
#### Post date: [November 3, 2021, 3:33pm UTC](https://discuss.ray.io/t/vtrace-understanding-shape-of-batches/4028/1 "2021-11-03T15:33:11Z")

</div>

Hi. I’m trying to understand how shapes of different batches of data are related in the VTrace Torch policy, in IMPALA.

In particular, how the train batch of observations is related to the actions and rewards batch.

In [`vtrace_torch_policy.py`](https://github.com/ray-project/ray/blob/e1e0cb5eaab4f1c7e9312f618a6b2e6d0712fc1d/rllib/agents/impala/vtrace_torch_policy.py#L133), the method [`build_vtrace_loss`](https://github.com/ray-project/ray/blob/e1e0cb5eaab4f1c7e9312f618a6b2e6d0712fc1d/rllib/agents/impala/vtrace_torch_policy.py#L133) contains the `train_batch`, a dictionary with a `'obs'` key, that is a batch of the agent observations while training (`train_batch['obs']` has shape `88 x SHAPE_OF_OBS_PER_TIMESTEP`).

I assume this 88 comes from the `rollout_fragment_length` parameter, which is set to 22.

Lets assume that `SHAPE_OF_OBS_PER_TIMESTEP` is 1, so each agent observation has 1 dimension, and the batch of observations is `88 x 1`.

Now, in `vtrace_torch.py`, there is the method [`from_importance_weights`](https://github.com/ray-project/ray/blob/e1e0cb5eaab4f1c7e9312f618a6b2e6d0712fc1d/rllib/agents/impala/vtrace_torch.py#L316) that contains something related to an estimation of the Q value:  
`deltas = clipped_rhos * (rewards + discounts * values_t_plus_1 - values)`

This `deltas` tensor is NOT the same shape as the `train_batch['obs']` from before, but instead has shape `21 x 4`.

So the batch of `rewards`, `values`, `discounts` and `values_t_plus_1` inside `vtrace_torch.py` has shape `21 x 4`, but the batch of observations (and actions and rewards) in `vtrace_torch_policy.py` have shape `88`.

I assume that, because 22 times 4 is 88, there is a “missing” value in the batch of `vtrace_torch.py`, but I’m not sure why, nor why the actual shape is 21 x 4: why 4 columns? is the “missing” value related to the first timestep? as we cannot estimate Q in time t=0, before having any observation?

My goal is to do some computation between the batchs from the two files (observations from one, Q estimates from the other), but having these different shapes, I don’t know how they are related to each other.