# Ray data.read\_csv keeps pausing

**URL:** <https://discuss.ray.io/t/ray-data-read-csv-keeps-pausing/12279>\
**Category:** Ray Data\
**Created:** [September 28, 2023, 1:02am UTC](https://discuss.ray.io/t/ray-data-read-csv-keeps-pausing/12279 "2023-09-28T01:02:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dhruva\_Kartik](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dhruva_kartik/32/5125_2.png) [@Dhruva\_Kartik](https://discuss.ray.io/u/Dhruva_Kartik)\
**Post date:** [September 28, 2023, 1:02am UTC](https://discuss.ray.io/t/ray-data-read-csv-keeps-pausing/12279/1 "2023-09-28T01:02:55Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

When I run this,

```auto
    dataset = ray.data.read_csv(
        s3_bucket_name+'/'+s3_folder_path_prefix+'/',
        partition_filter=partition_filter,
        filesystem=s3,
        convert_options=convert_options
    )

```

When I materialize this, it works but the read just keeps pausing or proceeds very slowly. But occasionally it goes forward in a burst. I was monitoring this process on top and I see that a lot of the processes are on sleep and only a few are running. Sometimes a lot of them start running and that is when the read progress bar moves forward.

I am using a 64 CPU instance and using all of them to read data. My data read function is very slow because of this. Is this an issue or is there some parameter that should be changed? A simple multiprocessing read function seems to be a lot faster than this.

---

<div class="post-metadata">

**Author:** ![Dhruva\_Kartik](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dhruva_kartik/32/5125_2.png) [@Dhruva\_Kartik](https://discuss.ray.io/u/Dhruva_Kartik)\
**Post date:** [September 28, 2023, 1:03am UTC](https://discuss.ray.io/t/ray-data-read-csv-keeps-pausing/12279/2 "2023-09-28T01:03:28Z")

</div>

![Screenshot 2023-09-27 at 5.54.24 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/f/f43a28b158c4c6d716e430169d39fa1d339f5eb4.jpeg)  
This is how top looks.

---

<div class="post-metadata">

**Author:** ![Dhruva\_Kartik](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dhruva_kartik/32/5125_2.png) [@Dhruva\_Kartik](https://discuss.ray.io/u/Dhruva_Kartik)\
**Post date:** [September 28, 2023, 1:04am UTC](https://discuss.ray.io/t/ray-data-read-csv-keeps-pausing/12279/3 "2023-09-28T01:04:37Z")

</div>

![Screenshot 2023-09-27 at 5.54.37 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/d/dd31e8899fcecaa802470eb5f5b531352b8e7043.jpeg)  
And sometimes, it is like this.

---

<div class="post-metadata">

**Author:** ![sjl](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sjl/32/6672_2.png) [@sjl](https://discuss.ray.io/u/sjl)\
**Post date:** [September 28, 2023, 8:36pm UTC](https://discuss.ray.io/t/ray-data-read-csv-keeps-pausing/12279/4 "2023-09-28T20:36:00Z")

</div>

Hi @Dhruva_Kartik , what is the intended use case for the dataset above (e.g. ingested by Trainer)? If you materialize the dataset, this will **read the entire dataset into memory** , which is often resource intensive and can cause failures (e.g. materializing a dataset which is significantly larger than memory available).

Since Ray Data provides [streaming execution](https://docs.ray.io/en/latest/data/data-internals.html#execution), it’s often times not necessary to materialize the dataset for most use cases. For example, if you want to iterate over the dataset, you can use `iter_batches()` without having to materialize the dataset.