# Map\_batches Optimization

**URL:** <https://discuss.ray.io/t/map-batches-optimization/23488>\
**Category:** Uncategorized\
**Created:** [February 24, 2026, 11:33am UTC](https://discuss.ray.io/t/map-batches-optimization/23488 "2026-02-24T11:33:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dgautam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dgautam/32/8503_2.png) [@dgautam](https://discuss.ray.io/u/dgautam)\
**Post date:** [February 24, 2026, 11:33am UTC](https://discuss.ray.io/t/map-batches-optimization/23488/1 "2026-02-24T11:33:22Z")

</div>

Hi, I am using ds.map\_batches(generate\_pdf\_batch, batch\_size=1000,batch\_format=“pandas”) but this is running on single head node on single core, and other all workers and core’s are in ideal state, can you please help

my code look like

try:  
for index,row in batch.iterrows():

```auto
    output_filename = f"/deepak/row_{row['twsss_id']}_Report.pdf"

    field_data = {
        'ucc': row['twsss_uniq_clnt_cd'],
        'nec': row['twsss_ldgr_bal_fr_fnds'],
        'tisin': row['twsss_id'],
        'tqs': row['twsss_ttl_qty_of_sec'],
        'tncom': row['twsss_ttl_no_of_com']
    }

    fillpdfs.write_fillable_pdf(' sejda.pdf',output_filename,field_data,flatten=True)

```

except Exception as e:  
print(f"An error occurred: {e}")  
return pd.DataFrame({‘status’: [‘done’] \* len(batch)})

ds = ray.data.from\_pandas(df)  
#ray.init(dashboard\_host=‘0.0.0.0’)  
results = ds.map\_batches(generate\_pdf\_batch, batch\_size=1000,batch\_format=“pandas”)

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [February 24, 2026, 11:33am UTC](https://discuss.ray.io/t/map-batches-optimization/23488/2 "2026-02-24T11:33:54Z")

</div>

Your code is only using the head node and a single core because, by default, Ray Data will use as many parallel tasks as there are blocks in your dataset. If your DataFrame is small or not partitioned, it will create only one block, resulting in a single task running on the head node. To utilize all workers and cores, repartition your dataset into more blocks before calling `map_batches`:

```python
ds = ray.data.from_pandas(df)
ds = ds.repartition(8) # Set to the number of available CPUs or nodes
results = ds.map_batches(generate_pdf_batch, batch_size=1000, batch_format="pandas")

```

This will split your dataset into multiple blocks, allowing Ray to schedule tasks across all available resources. For more details, see the discussion and solution in [Ray Discuss: Only head node is processing task](https://discuss.ray.io/t/only-head-node-is-processing-task/4782).

Would you like more detail on how to choose the number of partitions or how Ray schedules tasks?

Sources:

- [Ray Discuss: Only head node is processing task](https://discuss.ray.io/t/only-head-node-is-processing-task/4782)
- [Ray Data repartition usage](https://github.com/ray-project/ray/blob/master/python/ray/data/dataset.py)

Hint: Mention @RunLLM in the post for followups.

---

<div class="post-metadata">

**Author:** ![dgautam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dgautam/32/8503_2.png) [@dgautam](https://discuss.ray.io/u/dgautam)\
**Post date:** [February 24, 2026, 11:38am UTC](https://discuss.ray.io/t/map-batches-optimization/23488/3 "2026-02-24T11:38:20Z")

</div>

My dataset contain 1 cr records and i am using 32 core 3 workers with 16 core head node how many partition do i need and other things for more optimization