# Slow Large-Scale Ingest w/Ray AIR (Ray Data + Ray Train)

**URL:** <https://discuss.ray.io/t/slow-large-scale-ingest-w-ray-air-ray-data-ray-train/6949>\
**Category:** Uncategorized\
**Created:** [July 24, 2022, 11:30pm UTC](https://discuss.ray.io/t/slow-large-scale-ingest-w-ray-air-ray-data-ray-train/6949 "2022-07-24T23:30:27Z")\
**Posts on this page:** 1\
**Page:** 2

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [July 28, 2022, 12:28am UTC](https://discuss.ray.io/t/slow-large-scale-ingest-w-ray-air-ray-data-ray-train/6949/21 "2022-07-28T00:28:00Z")

</div>

Oh wow, this writeup is fantastic!

> The tasks per node seems to suggest that – unlike before – the scheduling might not happening according to the `SPREAD` strategy, as it seems that some workers are processing as much as 3.5x more tasks than others. I’m wondering if this might be the root cause of the slowdown and if so is there a way to specify that I want the `BatchMapper` to spread the workload?

To be extremely concise, the `BatchMapper` _should_ already be using SPREAD, but the problem is most like due to [this issue](https://github.com/ray-project/ray/issues/26878) which is being actively looked into.

> My batch delay seems a little high so maybe I’ll benefit from setting prefetch\_blocks=10 or something like that.

One thing to be aware of here is that prefetching blocks will utilize both memory and network transfer, so increasing this too much can poorly impact performance as well!

> Also, I’m currently setting the parallelism for my train dataset equal to the number of blocks in my train dataset (i.e. 2000), would decreasing this to say 500 potentially help reduce the number of tasks and speed up performance?

Hmm could you clarify this a bit? How many input files are you reading from? Decreasing the parallelism/number of blocks may certainly help with the issue you were seeing, but the caveat is that each block will be larger - depending on how large they are, this can potentially lead to OOM while running the preprocessing (though it looks like your setup has ample memory to support this).

[Previous page](https://discuss.ray.io/t/slow-large-scale-ingest-w-ray-air-ray-data-ray-train/6949.md?page=1)