# Ray datasets streaming block split?

**URL:** <https://discuss.ray.io/t/ray-datasets-streaming-block-split/11047>\
**Category:** Ray Data\
**Created:** [June 15, 2023, 10:50pm UTC](https://discuss.ray.io/t/ray-datasets-streaming-block-split/11047 "2023-06-15T22:50:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arik\_Mitschang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arik_mitschang/32/4546_2.png) [@Arik\_Mitschang](https://discuss.ray.io/u/Arik_Mitschang)\
**Post date:** [June 15, 2023, 10:50pm UTC](https://discuss.ray.io/t/ray-datasets-streaming-block-split/11047/1 "2023-06-15T22:50:09Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Low: It annoys or frustrates me for a moment.

Hi! I’m interested using ray for a data pipeline task that involves several somewhat long running steps and possibly indefinite stream of data. I think there are several options for the design, but using datasets seems to have an advantage in simplicity.

Because both long stream and long-running tasks, it is necessary to process to the end of the pipeline and write results regularly. Prior to 2.5.0, datasetpipelines were a way to accomplish this, and from this point on the streaming feature of datasets works similarly.

However, there is one sticking point I am wondering about: I can use batch\_size \> block\_size of map\_batches to gather up results which will effectively merge blocks and decrease parallelism (the task has overheads which make it more efficient per row given larger batches), but it doesn’t appear that I can split them again to increase parallelism in subsequent stages without repartition, which would require to materialize everything which would effectively disable the streaming functionality. datasetpipelines had a repartition\_each\_window which did something along these lines.

My questions are these: Is there an existing method to split blocks between map\_batches stages in a streaming fashion? If not, would such a feature violate the design principals or is it worth considering as a feature? Any other comments or things to consider?

Many thanks!

---

<div class="post-metadata">

**Author:** ![bveeramani](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bveeramani/32/2941_2.png) [@bveeramani](https://discuss.ray.io/u/bveeramani)\
**Post date:** [June 27, 2023, 12:29am UTC](https://discuss.ray.io/t/ray-datasets-streaming-block-split/11047/2 "2023-06-27T00:29:35Z")

</div>

Hey @Arik_Mitschang,

Streaming repartitions aren’t supported yet, but it’s a planned feature: [[data] [streaming] Support a streaming\_repartition() operator · Issue #36724 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/36724)