Hi folks, a few weeks ago, I tried to read json files from my HDFS database using the ray.data.read_json() API. But it was not successful. I checked the source code and it seems to me that it was indeed not supported. Can you confirm my statement? If it is supported, do you have any example code s…

Okay, I was able to read json file from HDFS. And below is my working code: from pyarrow import fs import ray from pprint import pprint from pyarrow import json import os CLASSPATH = os.popen('$HADOOP_HOME/bin/hadoop classpath --glob').read() ray.init(address='auto', runtime_env={"env_vars": {'…

Hi @Faaany , sorry for the issue, https://github.com/ray-project/ray/pull/37637 should fix the partition_filter error. Until next Ray release, you can use followed to work around: ray_data = ray.data.read_json(paths=path, filesystem=hdfs, partition_filter=None, read_options=read_options)

@chengsu Thanks for fixing the issue so quickly! But I got another problem running my code using a large jsonl file (~75GB). The data can be found here . For a small jsonl file (~106MB), the code worked out fine. Code: from pyarrow import fs import ray from pprint import pprint from pyarrow impo…

Does ray.data.read_json() support reading from HDFS?

Uncategorized

Faaany July 24, 2023, 3:47pm 5

close this issue, since it was fixed. Moved to another thread for further discussion: pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

Topic		Replies	Views
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays Uncategorized	0	2041	July 24, 2023
How to convert Pytorch torch.utils.data.Dataset to ray.data.dataset? Uncategorized	15	1637	December 8, 2022
[Dataset] Ray Dataset reading multiple parquet files with different columns crashes due to TProtocolException: Exceeded size limit Ray Data	14	2362	November 17, 2022
Loading large datasets from HDFS for xgboost on Yarn Ray Data	2	869	October 14, 2023
Ray data read hdfs slowly and process slowly Ray Train	3	578	August 31, 2023

Does ray.data.read_json() support reading from HDFS?

Related topics