# Read parquet files with wildcard/glob similar to Dask DataFrame?

**URL:** <https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423>\
**Category:** Ray Data\
**Created:** [April 28, 2023, 3:45pm UTC](https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423 "2023-04-28T15:45:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Scott\_Zelenka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/scott_zelenka/32/4328_2.png) [@Scott\_Zelenka](https://discuss.ray.io/u/Scott_Zelenka)\
**Post date:** [April 28, 2023, 3:45pm UTC](https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423/1 "2023-04-28T15:45:11Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I’m experimenting with moving some workloads from Dask to Ray. Rather than run the dask\_on\_ray helper, I’d like to learn how to do things natively in Ray. To start, I’m attempting to load a large collection of parquet files from GCS. In Dask we can do this with multiple wildcards in the path name, and it distributes the various parquet files across the cluster:

```auto
from dask.distributed import Client
import dask.dataframe as dd

client = Client("localhost:8786")

df = dd.read_parquet("gcs://bucket_name/prefix*/2023-03-01*.parquet")
df.head()

```

However, when using the same path with Ray, it complains about not being able to locate the Filename:

```auto
import ray

ray.init()
df = ray.data.read_parquet("gcs://bucket_name/prefix*/2023-03-01*.parquet")

```

> FileNotFoundError: gcs://bucket\_name/prefix\*/2023-03-01\*.parquet

If I reduce this to only load a single folder (without wildcards) within the bucket, it’ll scale and load in Ray, but it removes the ability to filter out files within the folder; or merge multiple datasets with a single call. This seems to be documented here:  
[https://docs.ray.io/en/latest/data/creating-datasets.html#reading-files-from-storage](https://docs.ray.io/en/latest/data/creating-datasets.html#reading-files-from-storage)

```auto
import ray

ray.init()
df = ray.data.read_parquet("gcs://bucket_name/prefix-full-folder-name/")

```

Is there a way to load files from storage using wildcards? Or what would be best practice be for loading data for a particular time range from a structure like this:

```auto
bucket_name:
  prefix-folder-one:
   2023-03-01.parquet
   2023-03-02.parquet
   2023-03-03.parquet
  prefix-folder-two:
   2023-03-01.parquet
   2023-03-02.parquet
   2023-03-03.parquet

```

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [May 1, 2023, 6:19pm UTC](https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423/2 "2023-05-01T18:19:08Z")

</div>

cc @chengsu to look at the question! I will move the question to the data channel.

---

<div class="post-metadata">

**Author:** ![Scott\_Zelenka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/scott_zelenka/32/4328_2.png) [@Scott\_Zelenka](https://discuss.ray.io/u/Scott_Zelenka)\
**Post date:** [May 2, 2023, 5:19pm UTC](https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423/3 "2023-05-02T17:19:09Z")

</div>

Creating the list before calling Ray does the trick, but adds a step when compared to Dask

```auto
import ray

# Create a GCS filesystem object
fs = gcsfs.GCSFileSystem()

# List all objects that match a wildcard string
files = fs.glob('gcs://bucket_name/prefix*/2023-03-01*.parquet')
files = list(map(lambda _: f"gcs://{_}", files))

ray.init()
df = ray.data.read_parquet(files)

```

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [May 11, 2023, 1:47am UTC](https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423/4 "2023-05-11T01:47:11Z")

</div>

cc @bveeramani @chengsu (seems like a data bug).

---

<div class="post-metadata">

**Author:** ![bveeramani](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bveeramani/32/2941_2.png) [@bveeramani](https://discuss.ray.io/u/bveeramani)\
**Post date:** [May 12, 2023, 12:47am UTC](https://discuss.ray.io/t/read-parquet-files-with-wildcard-glob-similar-to-dask-dataframe/10423/5 "2023-05-12T00:47:38Z")

</div>

I don’t think wildcard matching is a supported feature. @Scott_Zelenka could you file a feature request on GitHub?