# Map\_batches Optimization

**URL:** <https://discuss.ray.io/t/map-batches-optimization/23488>\
**Category:** Uncategorized\
**Created:** [February 24, 2026, 11:33am UTC](https://discuss.ray.io/t/map-batches-optimization/23488 "2026-02-24T11:33:22Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [February 24, 2026, 11:33am UTC](https://discuss.ray.io/t/map-batches-optimization/23488/2 "2026-02-24T11:33:54Z")

</div>

Your code is only using the head node and a single core because, by default, Ray Data will use as many parallel tasks as there are blocks in your dataset. If your DataFrame is small or not partitioned, it will create only one block, resulting in a single task running on the head node. To utilize all workers and cores, repartition your dataset into more blocks before calling `map_batches`:

```python
ds = ray.data.from_pandas(df)
ds = ds.repartition(8) # Set to the number of available CPUs or nodes
results = ds.map_batches(generate_pdf_batch, batch_size=1000, batch_format="pandas")

```

This will split your dataset into multiple blocks, allowing Ray to schedule tasks across all available resources. For more details, see the discussion and solution in [Ray Discuss: Only head node is processing task](https://discuss.ray.io/t/only-head-node-is-processing-task/4782).

Would you like more detail on how to choose the number of partitions or how Ray schedules tasks?

Sources:

- [Ray Discuss: Only head node is processing task](https://discuss.ray.io/t/only-head-node-is-processing-task/4782)
- [Ray Data repartition usage](https://github.com/ray-project/ray/blob/master/python/ray/data/dataset.py)

Hint: Mention @RunLLM in the post for followups.

---

_[View the full topic](https://discuss.ray.io/t/map-batches-optimization/23488)._