Hi folks, a few weeks ago, I tried to read json files from my HDFS database using the ray.data.read_json() API. But it was not successful. I checked the source code and it seems to me that it was indeed not supported. Can you confirm my statement? If it is supported, do you have any example code s…

Does ray.data.read_json() support reading from HDFS?

Faaany July 24, 2023, 3:47pm 5

close this issue, since it was fixed. Moved to another thread for further discussion: pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

Topic		Replies	Views
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays	0	1930	July 24, 2023
How to convert Pytorch torch.utils.data.Dataset to ray.data.dataset?	15	1413	December 8, 2022
[Dataset] Ray Dataset reading multiple parquet files with different columns crashes due to TProtocolException: Exceeded size limit Ray Data	14	1969	November 17, 2022
Read data from hdfs with ray Ray Core	8	1492	August 25, 2021
Loading large datasets from HDFS for xgboost on Yarn Ray Data	2	762	October 14, 2023