How can I create a ray dataset out of a pandas series containing bytes objects?

1

I’m trying to use ray.data.from_pandas to convert a pandas Series containing bytes objects to a ray dataset. I’d then like to map a custom function to this dataset that deserializes the bytes into a python object and then runs a method on that object.

But I get an error where some automatic casting takes place, which tells me that ray.data.from_pandas assumes in memory numpy arrays in the columns

ray.data.from_pandas(pandas_batch)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.10/site-packages/ray/data/read_api.py", line 2394, in from_pandas
    dfs = [_cast_ndarray_columns_to_tensor_extension(df.copy()) for df in dfs]
  File "/opt/conda/lib/python3.10/site-packages/ray/data/read_api.py", line 2394, in <listcomp>
    dfs = [_cast_ndarray_columns_to_tensor_extension(df.copy()) for df in dfs]
AttributeError: 'bytes' object has no attribute 'copy'

the pandas series I’m working with that has bytes elements

pandas_batch
0    b'\x02\r\x00\x00\x00outDbCoverage\x00\x00\x00\...
Name: _1, dtype: object

type(pandas_batch)
<class 'pandas.core.series.Series'>

Is there a different way I should be converting this pandas series to a ray dataset or do I need to roll my own data source? Any tips much appreciated!

2

I think this is working, I can at least create the dataset with

ray.data.from_items(pandas_batch.to_list())