When using a single-node cluster, how to efficiently share a dataframe (for read-only access) between ray actors/tasks?

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I have to calculate N metrics on a Pandas dataframe (accuracy, roc_auc, PR-curve etc). Ideally I would like each metric to be calculated in parallel using a different Ray actor/task. The dataframe itself can be large (10MM rows).

What is the best way to achieve this via Ray actors or tasks? Do I copy the dataframe to the object store via ray.put()?

At the moment, I am only using a single-node cluster, but hope to scale to multi-node clusters soon.

Thanks,
Abhishek

2

Yes, using ray.put() to store the dataframe in the object store, or passing the dataframe as arg in .remote() calls, should allow sharing the dataframe between different Ray tasks / actors with zero copy. The only caveat is that the dataframe cannot contain Python objects.

3

Consider using Datasets, which can load dataframes via ray.data.from_pandas(pandas_df). For example, to load a dataframe from pandas and split it into 100 blocks for parallel computation, do:

def batch_fn(batch: pd.DataFrame):
   return batch * 2

ds = ray.data.from_pandas(df)
ds = ds.repartition(100)
ds.map_batches(batch_fn)
4

Does python strings in Pandas dataset considered “Python objects”?
If yes which other string format could be chosen to speed up transfers?

5

Hi @anatolix thanks for raising the question!

Have you considered what @ericl proposed above? Indeed Ray Datasets is our recommended solution in this case. But I’m curious if you have specific requirements such that Ray Datasets wouldn’t work