# Large dataset ray dataset OOM

**URL:** <https://discuss.ray.io/t/large-dataset-ray-dataset-oom/11084>\
**Category:** Ray Tune\
**Created:** [June 19, 2023, 9:51am UTC](https://discuss.ray.io/t/large-dataset-ray-dataset-oom/11084 "2023-06-19T09:51:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ju\_Yunsang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ju_yunsang/32/4603_2.png) [@Ju\_Yunsang](https://discuss.ray.io/u/Ju_Yunsang)\
**Post date:** [June 19, 2023, 9:51am UTC](https://discuss.ray.io/t/large-dataset-ray-dataset-oom/11084/1 "2023-06-19T09:51:07Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- None: Just asking a question out of curiosity
- Low: It annoys or frustrates me for a moment.
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
- High: It blocks me to complete my task.

I have over 400G of big training data.  
I am managing with sqlite3 fotmat.  
Below is the code I test.  
OOM occurred during the preprocess and it is killed.  
It seems that OOM occurs by preprocessing the entire data and putting it in the object store before learning.  
Is there a way to preprocess every training iter and not upload that data in memory?

```auto
    ray_dataset = ray.data.read_sql(
            "SELECT * FROM documents LIMIT 10", create_connection
        )

    def _preprocess(batch):
        batch["total_text"] = batch[["title", "text"]].apply(
            lambda x: "{} {}".format(x[0] or "", x[1] or ""), axis=1
        )
        return batch.drop("title", axis=1).drop("text", axis=1)

    def _tokenize(batch):

        tokenized_output = tokenizer(
            batch["total_text"].values.tolist(),
            add_special_tokens=True,
            max_length=cfg.data.max_length,
            truncation=True,
            return_length=True,
            return_overflowing_tokens=True,
        )
        tokenized_output["labels"] = tokenized_output["input_ids"].clone()

        return tokenized_output

    preprocessing = BatchMapper(_preprocess, batch_format="pandas")
    tokenizing = BatchMapper(_tokenize, batch_format="pandas")
    preprocessor = Chain(preprocessing, tokenizing)

```

---

<div class="post-metadata">

**Author:** ![bveeramani](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bveeramani/32/2941_2.png) [@bveeramani](https://discuss.ray.io/u/bveeramani)\
**Post date:** [June 21, 2023, 7:55pm UTC](https://discuss.ray.io/t/large-dataset-ray-dataset-oom/11084/2 "2023-06-21T19:55:03Z")

</div>

Are you using an AIR `Trainer`? If so, could you try enabling [Streaming Ingest](https://docs.ray.io/en/latest/ray-air/check-ingest.html#enabling-streaming-ingest)?

---

<div class="post-metadata">

**Author:** ![Ju\_Yunsang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ju_yunsang/32/4603_2.png) [@Ju\_Yunsang](https://discuss.ray.io/u/Ju_Yunsang)\
**Post date:** [July 3, 2023, 1:52am UTC](https://discuss.ray.io/t/large-dataset-ray-dataset-oom/11084/3 "2023-07-03T01:52:02Z")

</div>

@bveeramani  
Thank you for reply  
i solved problem using [Configuring Training Datasets — Ray 2.5.1](https://docs.ray.io/en/latest/ray-air/check-ingest.html#enabling-streaming-ingest)