# Ray Dataset from\_generator equivalent

**URL:** <https://discuss.ray.io/t/ray-dataset-from-generator-equivalent/12448>\
**Category:** Ray Data\
**Created:** [October 13, 2023, 5:25pm UTC](https://discuss.ray.io/t/ray-dataset-from-generator-equivalent/12448 "2023-10-13T17:25:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nateyoder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nateyoder/32/4706_2.png) [@nateyoder](https://discuss.ray.io/u/nateyoder)\
**Post date:** [October 13, 2023, 5:25pm UTC](https://discuss.ray.io/t/ray-dataset-from-generator-equivalent/12448/1 "2023-10-13T17:25:15Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hello,

I have a use case where I was hoping to use ray to consume from a set of files where each file can fit into memory but the combination of them cannot. Unfortunately, these files are all stored in a custom format that cannot be read using pyarrow. I had been hoping to use something like [Tensorflow’s from\_generator](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_generator) function to do this but didn’t see any obvious way to do this from the current documentation.

Is this possible within Ray Data or would you have other suggestions on how to approach this task?

Thanks for your advice!

---

<div class="post-metadata">

**Author:** ![chengsu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chengsu/32/2922_2.png) [@chengsu](https://discuss.ray.io/u/chengsu)\
**Post date:** [October 18, 2023, 10:54pm UTC](https://discuss.ray.io/t/ray-dataset-from-generator-equivalent/12448/2 "2023-10-18T22:54:38Z")

</div>

Hi @nateyoder, if you know the files names in advance, you can do something like

```py
def read_file(file_name):
  # Your custom function to read the file in the custom format.
  data = ...
  return {"data": data}

file_names = ["foo", "bar", ...] # The list of file names.
ds = ray.data.from_items(file_names)
ds = ds.map(read_file)

```