# Avoid moving datasets around the network when using tune.with\_parameters

**URL:** <https://discuss.ray.io/t/avoid-moving-datasets-around-the-network-when-using-tune-with-parameters/15220>\
**Category:** Ray Tune\
**Created:** [July 13, 2024, 11:41am UTC](https://discuss.ray.io/t/avoid-moving-datasets-around-the-network-when-using-tune-with-parameters/15220 "2024-07-13T11:41:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hypersousage](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hypersousage/32/6315_2.png) [@hypersousage](https://discuss.ray.io/u/hypersousage)\
**Post date:** [July 13, 2024, 11:41am UTC](https://discuss.ray.io/t/avoid-moving-datasets-around-the-network-when-using-tune-with-parameters/15220/1 "2024-07-13T11:41:13Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I use ray.tune to tune the parameters of my models. I need to pass pretty large datasets (5 - 20 GB) to the trainable function and I am currently using tune.with\_parameters to pass the datasets.

This works fine, but each run requires the datasets to be transferred over the network to each of the nodes in the cluster. I don’t want to do this as it overloads the network and takes a while.

I already have copies of the datasets have on each server (a bunch of files on disk) and I want to read them from the local disk for each of the nodes. How to do this in ray? Also it is very important not to read files from disk for every trainable step.

---

<div class="post-metadata">

**Author:** ![sjl](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sjl/32/6672_2.png) [@sjl](https://discuss.ray.io/u/sjl)\
**Post date:** [July 17, 2024, 4:26pm UTC](https://discuss.ray.io/t/avoid-moving-datasets-around-the-network-when-using-tune-with-parameters/15220/2 "2024-07-17T16:26:10Z")

</div>

To avoid reading in the dataset for each trial, you can call `Dataset.materialize()` on the Ray Datasets prior to passing the data. This ensures that the dataset are materialized in memory, so that we are only passing around memory copies which are read per trial.

---

<div class="post-metadata">

**Author:** ![hypersousage](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hypersousage/32/6315_2.png) [@hypersousage](https://discuss.ray.io/u/hypersousage)\
**Post date:** [July 29, 2024, 12:07pm UTC](https://discuss.ray.io/t/avoid-moving-datasets-around-the-network-when-using-tune-with-parameters/15220/3 "2024-07-29T12:07:14Z")

</div>

This does not solve the problem with dataset transfer **between** servers