# Pytorch dataloader num\_workers with ray tune

**URL:** <https://discuss.ray.io/t/pytorch-dataloader-num-workers-with-ray-tune/21533>\
**Category:** RLlib\
**Created:** [January 20, 2025, 1:32pm UTC](https://discuss.ray.io/t/pytorch-dataloader-num-workers-with-ray-tune/21533 "2025-01-20T13:32:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jack\_caster](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jack\_caster](https://discuss.ray.io/u/jack_caster)\
**Post date:** [January 20, 2025, 1:32pm UTC](https://discuss.ray.io/t/pytorch-dataloader-num-workers-with-ray-tune/21533/1 "2025-01-20T13:32:00Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- None: Just asking a question out of curiosity

I started using Ray Tune for hyperparameters tuning and I got a prototype working. Now, I would like to understand a bit better how to allocate resources.

I have a pytorch model, which is trained on CPU (it is a little model with recursion, which does not benefit from a GPU). Currently, I assigned 1 CPU to each trial. The data is supplied to the model via a pytorch `dataloader`, which can be also parallelized setting `num_workers` (i.e., one trial can use `num_workers` to load the data). Without Ray Tune, I would set `num_workers = <n_cpus>`, but with Ray Tune already distributing trials across CPUs, I do not know what is the best value for `num_workers`.

Do you have any suggestions? Shall I set `num_workers = 1` or would it be ok also to set `num_workers = <n_cpus>`?

---

<div class="post-metadata">

**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\
**Post date:** [January 29, 2025, 12:15am UTC](https://discuss.ray.io/t/pytorch-dataloader-num-workers-with-ray-tune/21533/2 "2025-01-29T00:15:17Z")

</div>

Hi jack!  
So you’re correct `num_workers` works entirely on how many CPUs you have available. You can definitely increase it to `<n_cpus>` but overall I would try to not exceed `n_cpus` to overload the existing resources.

There’s a few docs on this that I’m gonna link here that might be of interest to you!

- [A Guide To Parallelism and Resources for Ray Tune — Ray 2.41.0](https://docs.ray.io/en/latest/tune/tutorials/tune-resources.html#a-guide-to-parallelism-and-resources-for-ray-tune)
- [Ray Data Internals — Ray 2.41.0](https://docs.ray.io/en/latest/data/data-internals.html#ray-data-and-tune)

---

<div class="post-metadata">

**Author:** ![ishaan-mehta](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ishaan-mehta/32/7822_2.png) [@ishaan-mehta](https://discuss.ray.io/u/ishaan-mehta)\
**Post date:** [May 6, 2025, 9:06pm UTC](https://discuss.ray.io/t/pytorch-dataloader-num-workers-with-ray-tune/21533/3 "2025-05-06T21:06:05Z")

</div>

Hi @christina, quick question for my understanding — if each trial is allotted 1 CPU, how can you set `num_workers` for the `DataLoader` to `<n_cpus>`? If we do that, aren’t we trying to use more resources than are available? For example if we run 4 trials on a worker node with 16 CPUs, then aren’t there only 12 logical CPUs available for the dataloaders?