# When using a single-node cluster, how to efficiently share a dataframe (for read-only access) between ray actors/tasks?

**URL:** <https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199>\
**Category:** Ray Core\
**Created:** [May 20, 2022, 6:21am UTC](https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199 "2022-05-20T06:21:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![AdamDivekar](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/adamdivekar/32/2556_2.png) [@AdamDivekar](https://discuss.ray.io/u/AdamDivekar)\
**Post date:** [May 20, 2022, 6:21am UTC](https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199/1 "2022-05-20T06:21:21Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

I have to calculate N metrics on a Pandas dataframe (accuracy, roc\_auc, PR-curve etc). Ideally I would like each metric to be calculated in parallel using a different Ray actor/task. The dataframe itself can be large (10MM rows).

What is the best way to achieve this via Ray actors or tasks? Do I copy the dataframe to the object store via `ray.put()`?

At the moment, I am only using a single-node cluster, but hope to scale to multi-node clusters soon.

Thanks,  
Abhishek

---

<div class="post-metadata">

**Author:** ![Mingwei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mingwei/32/997_2.png) [@Mingwei](https://discuss.ray.io/u/Mingwei)\
**Post date:** [May 20, 2022, 5:21pm UTC](https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199/2 "2022-05-20T17:21:10Z")

</div>

Yes, using `ray.put()` to store the dataframe in the object store, or passing the dataframe as arg in `.remote()` calls, should allow sharing the dataframe between different Ray tasks / actors with zero copy. The only caveat is that the dataframe cannot contain Python objects.

---

<div class="post-metadata">

**Author:** ![ericl](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ericl/32/133_2.png) [@ericl](https://discuss.ray.io/u/ericl)\
**Post date:** [May 20, 2022, 9:29pm UTC](https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199/3 "2022-05-20T21:29:33Z")

</div>

Consider using [Datasets](https://docs.ray.io/en/latest/data/dataset.html), which can load dataframes via `ray.data.from_pandas(pandas_df)`. For example, to load a dataframe from pandas and split it into 100 blocks for parallel computation, do:

```auto
def batch_fn(batch: pd.DataFrame):
   return batch * 2

ds = ray.data.from_pandas(df)
ds = ds.repartition(100)
ds.map_batches(batch_fn)

```

---

<div class="post-metadata">

**Author:** ![anatolix](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/anatolix/32/3284_2.png) [@anatolix](https://discuss.ray.io/u/anatolix)\
**Post date:** [October 6, 2022, 12:42pm UTC](https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199/4 "2022-10-06T12:42:15Z")

</div>

> [@Mingwei](#):
>
> The only caveat is that the dataframe cannot contain Python objects.

Does python strings in Pandas dataset considered “Python objects”?  
If yes which other string format could be chosen to speed up transfers?

---

<div class="post-metadata">

**Author:** ![zhz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zhz/32/12_2.png) [@zhz](https://discuss.ray.io/u/zhz)\
**Post date:** [December 2, 2022, 1:11am UTC](https://discuss.ray.io/t/when-using-a-single-node-cluster-how-to-efficiently-share-a-dataframe-for-read-only-access-between-ray-actors-tasks/6199/5 "2022-12-02T01:11:58Z")

</div>

Hi @anatolix thanks for raising the question!

Have you considered what @ericl proposed above? Indeed Ray Datasets is our recommended solution in this case. But I’m curious if you have specific requirements such that Ray Datasets wouldn’t work