# How to stream data directly from s3

**URL:** <https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838>\
**Category:** Ray Train\
**Created:** [February 26, 2024, 3:55pm UTC](https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838 "2024-02-26T15:55:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aleksei](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@Aleksei](https://discuss.ray.io/u/Aleksei)\
**Post date:** [February 26, 2024, 3:55pm UTC](https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838/1 "2024-02-26T15:55:52Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

We need to train pytorch models on large datasets 500GB-2TB.  
We are going to use ray data+train, so currently we are designing our solution.  
It seems that we need pipelining from s3 because

1. Allocating 500GB-2TB of memory for storing dataset in global storage would be expensive
2. Enabling spilling would affect performance.
3. S3 speed could be boosted up to 10 GB/s which is faster than any disk and could be compared to training from memory

So it looks like training on data streamed directly from s3 without storing the whole dataset on disk/memory/GCS would be an optimal solution.

Does ray support this mode?  
Can you help me with where to start to do that?

---

<div class="post-metadata">

**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\
**Post date:** [February 29, 2024, 1:35am UTC](https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838/2 "2024-02-29T01:35:40Z")

</div>

Yes! We recommend you to use Ray Data + Ray Train to stream data directly from S3 and split it to your distributed training workers on the fly.

See this user guide: [Data Loading and Preprocessing — Ray 2.9.3](https://docs.ray.io/en/latest/train/user-guides/data-loading-preprocessing.html)

See here for how to load data from s3: [Loading Data — Ray 2.9.3](https://docs.ray.io/en/latest/data/loading-data.html#reading-files-from-cloud-storage)

Let me know if you run into any problems!

---

<div class="post-metadata">

**Author:** ![Aleksei](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@Aleksei](https://discuss.ray.io/u/Aleksei)\
**Post date:** [March 4, 2024, 10:59am UTC](https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838/3 "2024-03-04T10:59:14Z")

</div>

Thank you, it works as expected.