# Best practices around handling giant datasets with ray data (large amount of read tasks)

**URL:** <https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809>\
**Category:** Uncategorized\
**Created:** [September 15, 2024, 4:20am UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809 "2024-09-15T04:20:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![eric](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eric/32/6319_2.png) [@eric](https://discuss.ray.io/u/eric)\
**Post date:** [September 15, 2024, 4:20am UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809/1 "2024-09-15T04:20:23Z")

</div>

We’ve implemented a custom datasource that reads from an index in postgres and downloads image files from s3.

We are working with a 300+ million scale dataset. The datasource currently reports the correct dataset size for the entire dataset. Thus, Ray decides to split it into many read tasks (1 million+).

This causes a huge delay (30 minutes +) during startup (likely due to pickling the read tasks themselves). We did everything to make the overall pickled read\_fn as tiny as possible (`_open_block` is a top-level function within the module):

```auto
    def _create_read_fn(self, offset: int, limit: int):
        table_name = self.table_name

        def read_fn():
            yield _open_block(table_name, offset, limit)

        return read_fn

```

Is there a more efficient way to handle this?

---

<div class="post-metadata">

**Author:** ![bveeramani](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bveeramani/32/2941_2.png) [@bveeramani](https://discuss.ray.io/u/bveeramani)\
**Post date:** [September 18, 2024, 7:16pm UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809/2 "2024-09-18T19:16:19Z")

</div>

Hey @eric, what happens if you configure `override_num_blocks` and set it to a (relatively) low value?

---

<div class="post-metadata">

**Author:** ![eric](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eric/32/6319_2.png) [@eric](https://discuss.ray.io/u/eric)\
**Post date:** [September 19, 2024, 3:54am UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809/3 "2024-09-19T03:54:27Z")

</div>

It spills to disk and eventually runs out of disk space

---

<div class="post-metadata">

**Author:** ![eric](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eric/32/6319_2.png) [@eric](https://discuss.ray.io/u/eric)\
**Post date:** [October 8, 2024, 9:31pm UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809/4 "2024-10-08T21:31:47Z")

</div>

@bveeramani is this to be expected? From my understanding ray read tasks are always executed fully?

---

<div class="post-metadata">

**Author:** ![bveeramani](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bveeramani/32/2941_2.png) [@bveeramani](https://discuss.ray.io/u/bveeramani)\
**Post date:** [October 8, 2024, 9:49pm UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809/5 "2024-10-08T21:49:37Z")

</div>

No, this isn’t expected.

What version of Ray are you using?

> From my understanding ray read tasks are always executed fully?

Not quite. The Ray Data scheduler should only pull data from a read task if there’s space available in the object store. So, if a read task yields 128 MiB batches and the object store is almost full, it shouldn’t continue executing the task until space is freed.

---

<div class="post-metadata">

**Author:** ![eric](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eric/32/6319_2.png) [@eric](https://discuss.ray.io/u/eric)\
**Post date:** [October 15, 2024, 6:48am UTC](https://discuss.ray.io/t/best-practices-around-handling-giant-datasets-with-ray-data-large-amount-of-read-tasks/15809/6 "2024-10-15T06:48:57Z")

</div>

Ahh good to know. Seems like fundamentally [[core] Correctly report size of variable shaped TensorArray by BitPhinix · Pull Request #47743 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/47743) was the issue