# Benchmarks for Ray Data? **URL:** \ **Category:** Ray Data\ **Created:** [October 12, 2022, 12:50pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875 "2022-10-12T12:50:51Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![jinnovation](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jinnovation/32/3296_2.png) [@jinnovation](https://discuss.ray.io/u/jinnovation)\ **Post date:** [October 12, 2022, 12:50pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/1 "2022-10-12T12:50:51Z")
Similar to my other question regarding [benchmarks for Ray Serve](https://discuss.ray.io/t/benchmarks-for-ray-serve/7858), are there benchmarks that have either been published or in the works for [Ray Data](https://docs.ray.io/en/latest/data/dataset.html) in comparison to TF Transform, Dataflow, or other preprocessing solutions? ---
**Author:** ![Jiao\_Dong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jiao_dong/32/1821_2.png) [@Jiao\_Dong](https://discuss.ray.io/u/Jiao_Dong)\ **Post date:** [October 12, 2022, 6:00pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/2 "2022-10-12T18:00:40Z")
Hi @jinnovation great question and right time. Benchmarking is one of key focus area of Ray AIR w/ Ray Dataset in Ray 2.1 & 2.2 that we’re actively testing and improving on weekly basis. You should hear from us soon 🙂 ---
**Author:** ![loneystar1983](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/loneystar1983/32/3731_2.png) [@loneystar1983](https://discuss.ray.io/u/loneystar1983)\ **Post date:** [January 17, 2023, 2:17am UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/3 "2023-01-17T02:17:57Z")
Hi @Jiao_Dong, is there any benchmark result of Ray Dataset now? ---
**Author:** ![zhz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zhz/32/12_2.png) [@zhz](https://discuss.ray.io/u/zhz)\ **Post date:** [January 18, 2023, 5:31pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/4 "2023-01-18T17:31:35Z")
Hi @jinnovation @loneystar1983 thanks for your interests in Ray Data! We are actively working on this. For now, we have some results from the two most recent Ray Enhancement Proposals: - [Ray Data Execution optimizer](https://github.com/ray-project/enhancements/blob/main/reps/2022-12-15-optimizer-data.md) - [Ray Data native pipelined execution](https://github.com/ray-project/enhancements/blob/main/reps/2022-12-14-native-pipelining-data.md) ---
**Author:** ![loneystar1983](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/loneystar1983/32/3731_2.png) [@loneystar1983](https://discuss.ray.io/u/loneystar1983)\ **Post date:** [January 28, 2023, 2:26am UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/5 "2023-01-28T02:26:50Z")
Thanks @zhz ! These proposals are greatly helpful, I can preprocess my data more fastly. And more, is there a comparision between Ray Data and Spark DataFrame? ---
**Author:** ![zhz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zhz/32/12_2.png) [@zhz](https://discuss.ray.io/u/zhz)\ **Post date:** [February 28, 2023, 6:01am UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/6 "2023-02-28T06:01:08Z")
Some benchmark available here! [Developer Preview: Ray Data Streaming Execution - Google Docs](https://docs.google.com/document/d/1BXd1cGexDnqHAIVoxTnV3BV0sklO9UXqPwSdHukExhY/edit#heading=h.dkdm0ffungpx) We are working on a benchmark study that involves Spark DataFrames and will share the results in 3~5 weeks ---
**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\ **Post date:** [March 14, 2023, 6:06pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/7 "2023-03-14T18:06:44Z")
@jinnovation Stay tuned. We will be publishing these benchmarks as blogs in the coming weeks. Can you close this issue since we are in the process sharing our findings with the community? ---
**Author:** ![zhz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zhz/32/12_2.png) [@zhz](https://discuss.ray.io/u/zhz)\ **Post date:** [September 21, 2023, 4:38pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/8 "2023-09-21T16:38:40Z")
Hi @jinnovation please checkout this blog: [Fast, flexible, and scalable data loading for ML training with Ray Data](https://www.anyscale.com/blog/fast-flexible-scalable-data-loading-for-ml-training-with-ray-data) ---
**Author:** ![Li\_Bin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/li_bin/32/4051_2.png) [@Li\_Bin](https://discuss.ray.io/u/Li_Bin)\ **Post date:** [September 23, 2023, 1:49pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/9 "2023-09-23T13:49:18Z")
Here is my case. I have 200 image files and the image in each file is of big size. I implement a datasource to read those files with a parallelism number following the [ray.io](http://ray.io) doc. When I try to ingest this dataset to a prepocessor and then the training worker in a streaming way , my object store is out of memory from time to time. The trainning worker consums the output of the preprocessor very slow but I can’t control the datasource loading speed. As the buffer, the object store oveflows. Is there any way to cascade the flow control to the data source? That is, suppose the buffer size is fixed , when the data sink is slow, just slow down the data incoming. Any help will be great. ---
**Author:** ![sjl](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sjl/32/6672_2.png) [@sjl](https://discuss.ray.io/u/sjl)\ **Post date:** [September 25, 2023, 5:34pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/10 "2023-09-25T17:34:06Z")
Hi @Li_Bin , what version of ray are you using? We recommend directly using [ray.data.read\_images](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_images.html) to read the image files. With Ray 2.7, you get both streaming execution and auto parallelism detection, which should help address both of the concerns you mention above. Let me know if this is helpful, or if you have any further questions. ---
**Author:** ![Li\_Bin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/li_bin/32/4051_2.png) [@Li\_Bin](https://discuss.ray.io/u/Li_Bin)\ **Post date:** [September 26, 2023, 12:07am UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/11 "2023-09-26T00:07:36Z")
I am using 2.7. My image files have specific format other than those supported by read\_images. I implement my own datasource with yield. Let me ask this question from other way: Suppose my dataset sink ,say, the traininer , consumes bolcks very very very slow, what will happen? The image reading from disk will be blocked and not accept more new data into the pipeline or the new data continues to come into the heap/object store memory to buffer? ---
**Author:** ![sjl](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sjl/32/6672_2.png) [@sjl](https://discuss.ray.io/u/sjl)\ **Post date:** [September 28, 2023, 8:27pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/12 "2023-09-28T20:27:23Z")
Got it, thanks for rephrasing the question, I think I understand what you are getting at. In the above scenario, as you observed, Ray Data will launch multiple read tasks to load data into the pipeline, even if the trainer only consumes 1 image at a time in a very slow manner. This is likely the reason you are seeing the object store OOM. Ray Data has some backpressure logic which kicks in to limit tasks which are not needed, but it’s likely that the large file size you mention is causing an issue here. We are working to address this bug in a future release, likely by the end of the year. In the meantime, a potential workaround would be to increase the number of CPUs for read tasks, so that it limits the number of read tasks scheduled. For example: ```auto ray.data.read_datasource(..., ray_remote_args={"num_cpus": 5},) ``` Let me know if this is helpful @Li_Bin ---
**Author:** ![Li\_Bin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/li_bin/32/4051_2.png) [@Li\_Bin](https://discuss.ray.io/u/Li_Bin)\ **Post date:** [October 4, 2023, 2:40pm UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/13 "2023-10-04T14:40:11Z")
Thanks for your workaround . I will try . ---
**Author:** ![Li\_Bin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/li_bin/32/4051_2.png) [@Li\_Bin](https://discuss.ray.io/u/Li_Bin)\ **Post date:** [October 5, 2023, 1:34am UTC](https://discuss.ray.io/t/benchmarks-for-ray-data/7875/14 "2023-10-05T01:34:28Z")
hi, To reprocude the issues I met, I work out a exmaple as below : [https://colab.research.google.com/drive/1fTbeuCIN1Ojhok46Qu7SMhpNtiGW7taM?usp=sharing](https://colab.research.google.com/drive/1fTbeuCIN1Ojhok46Qu7SMhpNtiGW7taM?usp=sharing) From the example, I have serveral questions : 1). When I set the PARALLELISM= -1, suppose I have 41 files and the parallelism determined by the system is 200. System says: “To satisfy the requested parallelism of 200, each read task output is split into 5 smaller blocks.” . My quesion is how to split the single bolck which actually is a single file to 5 smaller blocks? 2) No matter what number I set the PARALLELISM , the dataset.train\_test\_split function will load all the images into memory and not a lazy execution? 3). For the scenario where the resource is very limited, for example the colab with 2 CPUs, 1GPU, small RAM, the training will be OOM anyway. The workaroud given seems not to work When I set the ray\_remote\_args={“num\_cpus”: 2} . 4) One more further question is : Suppose I will finally feed ,say, 10G data into one GPU, How can I estimate the memory size used in heap and in object store at each stage from the streaming pipeline backwards (suppose I have several preprocessor maps) ? Thanks a lot.