I am trying to read a huge parquet dataset which is partitioned. Is it possible to use ray.data.read_parquet to read this dataset only to read a few selected partitions?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Ray Dataset Cannot Read Parquet File | 1 | 665 | August 1, 2022 | |
|
Why isn't `ray.data.read_api._get_reader` parallelized?
|
0 | 195 | December 5, 2023 | |
|
Ray Data read Parquet loads all the data in one go
|
4 | 633 | October 21, 2023 | |
|
Hive Partitioned Datasets
|
0 | 468 | July 3, 2023 | |
| Data loading of parquet files is very memory consuming | 2 | 1451 | June 21, 2022 |