# Error after shuffeling ray dataset when splitting in train und test

**URL:** <https://discuss.ray.io/t/error-after-shuffeling-ray-dataset-when-splitting-in-train-und-test/8437>\
**Category:** Ray Data\
**Created:** [November 25, 2022, 5:55pm UTC](https://discuss.ray.io/t/error-after-shuffeling-ray-dataset-when-splitting-in-train-und-test/8437 "2022-11-25T17:55:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alpe6825](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alpe6825/32/3214_2.png) [@Alpe6825](https://discuss.ray.io/u/Alpe6825)\
**Post date:** [November 25, 2022, 5:55pm UTC](https://discuss.ray.io/t/error-after-shuffeling-ray-dataset-when-splitting-in-train-und-test/8437/1 "2022-11-25T17:55:40Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

**The Problem:**

I have a large ray dataset and want to split it like this:

```auto
train_ds, test_ds = ray_ds.train_test_split(test_size=0.05, shuffle=True)

for batch in train.iter_torch_batches(batch_size=1, device="cuda:0", ):
      # training

```

Unfortunately it only works with `shuffle=False` otherwise I get the following error:

```auto
Traceback (most recent call last):
  File "/home/USER/PycharmProjects/Projectname_Preprocessing/CanonicalDataset.py", line 395, in <module>
    for batch in train.iter_torch_batches(batch_size=1, device="cuda:0", ):
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/data/dataset.py", line 2523, in iter_torch_batches
    for batch in self.iter_batches(
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/data/dataset.py", line 2450, in iter_batches
    yield from batch_blocks(
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/data/_internal/block_batching.py", line 129, in batch_blocks
    yield from get_batches(block_window[0])
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/data/_internal/block_batching.py", line 99, in get_batches
    result = _format_batch(batch, batch_format)
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/data/_internal/block_batching.py", line 147, in _format_batch
    batch = BlockAccessor.for_block(batch).to_numpy()
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/data/_internal/arrow_block.py", line 217, in to_numpy
    arrays.append(array.to_numpy(zero_copy_only=False))
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/air/util/tensor_extensions/arrow.py", line 285, in to_numpy
    return self._to_numpy(zero_copy_only=zero_copy_only)
  File "/home/USER/anaconda3/envs/Projectname_Preprocessing/lib/python3.9/site-packages/ray/air/util/tensor_extensions/arrow.py", line 269, in _to_numpy
    return np.ndarray(shape, dtype=ext_dtype, buffer=data_buffer, offset=offset)
TypeError: buffer is too small for requested array

```

I’m grateful for any ideas how to fix it.

---

<div class="post-metadata">

**Author:** ![Clark\_Zinzow](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/clark_zinzow/32/445_2.png) [@Clark\_Zinzow](https://discuss.ray.io/u/Clark_Zinzow)\
**Post date:** [November 29, 2022, 7:27pm UTC](https://discuss.ray.io/t/error-after-shuffeling-ray-dataset-when-splitting-in-train-und-test/8437/2 "2022-11-29T19:27:50Z")

</div>

Hi @Alpe6825, what version of Ray are you using? I think we fixed a similar issue in a recent release (or in latest master).

---

<div class="post-metadata">

**Author:** ![Alpe6825](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alpe6825/32/3214_2.png) [@Alpe6825](https://discuss.ray.io/u/Alpe6825)\
**Post date:** [November 29, 2022, 10:46pm UTC](https://discuss.ray.io/t/error-after-shuffeling-ray-dataset-when-splitting-in-train-und-test/8437/3 "2022-11-29T22:46:09Z")

</div>

Hi @Clark_Zinzow. Thank you for your reply. I use ray 2.1.0 installed via pip.