Hi folks, a few weeks ago, I tried to read json files from my HDFS database using the ray.data.read_json() API. But it was not successful. I checked the source code and it seems to me that it was indeed not supported. Can you confirm my statement? If it is supported, do you have any example code s…

Does ray.data.read_json() support reading from HDFS?

Faaany July 24, 2023, 3:47pm 5

close this issue, since it was fixed. Moved to another thread for further discussion: pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

Topic		Replies	Views
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays	0	1947	July 24, 2023
How to convert Pytorch torch.utils.data.Dataset to ray.data.dataset?	15	1444	December 8, 2022
[Dataset] Ray Dataset reading multiple parquet files with different columns crashes due to TProtocolException: Exceeded size limit Ray Data	14	2049	November 17, 2022
Read data from hdfs with ray Ray Core	8	1497	August 25, 2021
Loading large datasets from HDFS for xgboost on Yarn Ray Data	2	767	October 14, 2023