# Metadata fetching seems to be a sequential run

**URL:** <https://discuss.ray.io/t/metadata-fetching-seems-to-be-a-sequential-run/20544>\
**Category:** Ray Data\
**Created:** [November 12, 2024, 5:24am UTC](https://discuss.ray.io/t/metadata-fetching-seems-to-be-a-sequential-run/20544 "2024-11-12T05:24:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bryany](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bryany/32/7301_2.png) [@bryany](https://discuss.ray.io/u/bryany)\
**Post date:** [November 12, 2024, 5:24am UTC](https://discuss.ray.io/t/metadata-fetching-seems-to-be-a-sequential-run/20544/1 "2024-11-12T05:24:43Z")

</div>

I have 10k small files (each file size is around 2MB) located on s3, the reading process of ray data seems very slow at the metadata fetching stage, from the logs it seems it’s reading data in sequential.

Can this run be parallelized? Otherwise it looks to me pretty awkward to spend 10 minutes reading the metadata only.

```auto
Metadata Fetch Progress 0: 100%|██████████| 4.18k/4.18k [02:53<00:00, 24.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.18k/4.18k [02:54<00:00, 24.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.18k/4.18k [02:55<00:00, 24.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.18k/4.18k [02:56<00:00, 24.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.18k/4.18k [02:57<00:00, 24.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.18k/4.18k [02:58<00:00, 24.6 task/s]
Metadata Fetch Progress 0: 99%|█████████▉| 4.18k/4.21k [02:59<00:01, 24.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.21k/4.21k [02:59<00:00, 16.0 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.21k/4.21k [02:59<00:00, 16.0 task/s]
Metadata Fetch Progress 0: 97%|█████████▋| 4.21k/4.33k [03:00<00:07, 16.0 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.33k/4.33k [03:00<00:00, 25.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.33k/4.33k [03:00<00:00, 25.6 task/s]
Metadata Fetch Progress 0: 98%|█████████▊| 4.33k/4.42k [03:01<00:03, 25.6 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:01<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:01<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:02<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:03<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:04<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:05<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.42k/4.42k [03:06<00:00, 33.1 task/s]
Metadata Fetch Progress 0: 99%|█████████▊| 4.42k/4.48k [03:07<00:01, 33.1 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.48k/4.48k [03:07<00:00, 21.9 task/s]
Metadata Fetch Progress 0: 100%|██████████| 4.48k/4.48k [03:07<00:00, 21.9 task/s]

```

---

<div class="post-metadata">

**Author:** ![Nano1337](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nano1337/32/7721_2.png) [@Nano1337](https://discuss.ray.io/u/Nano1337)\
**Post date:** [March 1, 2025, 5:47am UTC](https://discuss.ray.io/t/metadata-fetching-seems-to-be-a-sequential-run/20544/2 "2025-03-01T05:47:01Z")

</div>

ray.data has an API called `read_parquet_bulk()` which takes in a list of parquet files (not a directory). You can run the following instead:

```auto
        import s3fs
        fs = s3fs.S3FileSystem()
        input_ds: Dataset = read_parquet_bulk(
            [
                "s3://" + f
                for f in fs.glob(f"{relevant_directory}/*.parquet")
            ],
        )

```

This cut down my metadata fetch time from 11 minutes to 18 seconds. Hope this helps!