Read data from hdfs with ray

Ray Core
1

hi ,
i want to read data from hdfs but i got error that was Unable to load libhdfs . ı used modin . What can i do ?
thank you

2

cc @Alex Maybe good Dataset use case?

3

Maybe try Datasets: Distributed Arrow on Ray — Ray v2.0.0.dev0

You can pass in filesystem= to the read APIs to specify a Hadoop pyarrow filesystem: pyarrow.fs.HadoopFileSystem — Apache Arrow v5.0.0

4

hi @ericl ,
should ı install ray v2.0.0.dev0 ?

ı used arroy but ı got error .

AttributeError: module ‘pickle’ has no attribute ‘PickleBuffer’

5

cc @Alex can you answer what’s the best action to get around his issue?

6

Hey @murat, I don’t have an hdfs cluster handy, but what version of pyarrow do you have (pip freeze | grep pyarrow)?

7

pip install cloudpickle should fix pickle error.

I think, it would be nice to include requirements.txt for Ray.data module.

8

@Alex @sangcho would it be possible to get pinged when bug in Modin pops up here in the future?

9

@devin-petersohn will do!