# Shared dataset on a local desktop

**URL:** <https://discuss.ray.io/t/shared-dataset-on-a-local-desktop/9189>\
**Category:** Uncategorized\
**Created:** [February 1, 2023, 4:14pm UTC](https://discuss.ray.io/t/shared-dataset-on-a-local-desktop/9189 "2023-02-01T16:14:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![lthiet](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lthiet/32/3742_2.png) [@lthiet](https://discuss.ray.io/u/lthiet)\
**Post date:** [February 1, 2023, 4:14pm UTC](https://discuss.ray.io/t/shared-dataset-on-a-local-desktop/9189/1 "2023-02-01T16:14:05Z")

</div>

Hello,

I am currently using Ray as a way to find hyperparameters for my model. My setup is a desktop computer that has enough VRAM to host 4 models, but not enough RAM to store 4 instances of the dataset.

I am trying to use this snippet: [Ray Tune FAQ — Ray 2.43.0](https://docs.ray.io/en/latest/tune/faq.html#how-can-i-use-large-datasets-in-tune) in order to have a single dataset where the reference is shared accross each trials that are running on my PC. Unfortunately, memory usage seems to increase too much anyway and the script crashes, despite knowing that the dataset object has only been instanced once.

You can find my script here: [from ray import air, tunefrom absl import appfrom absl import flagsfrom ty - Pastebin.com](https://pastebin.com/zQbA1wT4) relevant LOC are 76, 83, 38

What am I doing wrong? Is there another way to achieve what I am describing?

Many thanks!

---

<div class="post-metadata">

**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\
**Post date:** [March 7, 2023, 12:03am UTC](https://discuss.ray.io/t/shared-dataset-on-a-local-desktop/9189/2 "2023-03-07T00:03:25Z")

</div>

Hi @lthiet,

Ray Data might be what you’re looking for! Ray Data supports pipelined (lazy loading and preprocessing) of data. You can ingest the data through `ray.data.read_csv`, `ray.data.read_parquet`, etc. See [Getting Started — Ray 2.3.0](https://docs.ray.io/en/latest/data/getting-started.html) for more info.

This way, you won’t need to store the entire dataset in memory all at once before passing it into training. Under the hood, Ray Data stores dataset blocks in memory across all nodes in your cluster with the Ray Object Store.

You’ll want to use the new Dataset Streaming API introduced in 2.3.0! See here for more info and some snippets you can build on: [Developer Preview: Ray Data Streaming Execution - Google Docs](https://docs.google.com/document/d/1BXd1cGexDnqHAIVoxTnV3BV0sklO9UXqPwSdHukExhY/edit#)

Once you’ve configured Ray Data ingest, you can pass in the Ray Dataset to Tune and start using it:

```python
train_ds = ray.data.read_parquet(...)

def train_fn(config, train_ds=None):
    for batch in train_ds.iter_batches(batch_size=32):
        pass # do training on the batch

tuner = Tuner(tune.with_parameters(train_fn, train_ds=train_ds))
tuner.fit()

```

Let me know if this works for you, and also be sure to give feedback to the Ray Data team as mentioned in the “Feedback Wanted” section on that doc.