# Cannot read parquet files

**URL:** <https://discuss.ray.io/t/cannot-read-parquet-files/10209>\
**Category:** Ray Data\
**Created:** [April 12, 2023, 8:03pm UTC](https://discuss.ray.io/t/cannot-read-parquet-files/10209 "2023-04-12T20:03:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![skunkwerk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/skunkwerk/32/4235_2.png) [@skunkwerk](https://discuss.ray.io/u/skunkwerk)\
**Post date:** [April 12, 2023, 8:03pm UTC](https://discuss.ray.io/t/cannot-read-parquet-files/10209/1 "2023-04-12T20:03:48Z")

</div>

I have 15 Parquet files I’m trying to load into Ray, but I’m getting this error:

The blocks of this dataset are estimated to be 15.8x larger than the target block size of 512 MiB. This may lead to out-of-memory errors during processing. Consider reducing the size of input files or using `.repartition(n)` to increase the number of dataset blocks.

And indeed my Ray workers die:  
1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node

My code is:

```auto
files = glob.glob('/download_folder/*')
dataset = ray.data.read_parquet(files)

```

I then attempted to repartition the data like this code, but it made no difference:

```auto
size = dataset.size_bytes()
size_mb = size * 0.000001
ideal_number_blocks = int(size_mb / 512)
logger.debug(f"repartitioning into {ideal_number_blocks} blocks")
dataset = dataset.repartition(ideal_number_blocks)

```

When loaded using PyArrow, I can see that a sample file has this metadata:  
num\_columns: 241  
num\_rows: 4770085  
num\_row\_groups: 13  
serialized\_size: 408988

the row groups seem to have a batch size of around 164 MB each:  
\<pyarrow.\_parquet.RowGroupMetaData object at 0x7f7dc27bff40\>  
num\_columns: 241  
num\_rows: 395374  
total\_byte\_size: 164240468

How do I fix this?

thanks

---

<div class="post-metadata">

**Author:** ![jianxiao](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@jianxiao](https://discuss.ray.io/u/jianxiao)\
**Post date:** [April 13, 2023, 8:43pm UTC](https://discuss.ray.io/t/cannot-read-parquet-files/10209/2 "2023-04-13T20:43:32Z")

</div>

Hi @skunkwerk Which Ray version are you using? How many nodes do you use in the Ray cluster and what’s spec look like?

Note: that Parquet files are very well compressed and can be a few times larger when decompressed when loading into Ray.

---

<div class="post-metadata">

**Author:** ![skunkwerk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/skunkwerk/32/4235_2.png) [@skunkwerk](https://discuss.ray.io/u/skunkwerk)\
**Post date:** [April 19, 2023, 9:11pm UTC](https://discuss.ray.io/t/cannot-read-parquet-files/10209/3 "2023-04-19T21:11:12Z")

</div>

I’m using Ray 2.3.1, on a single local node with 64 GB of RAM, 4 CPUs.

thanks