Pytorch dataloader num_workers with ray tune

RLlib
1

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity

I started using Ray Tune for hyperparameters tuning and I got a prototype working. Now, I would like to understand a bit better how to allocate resources.

I have a pytorch model, which is trained on CPU (it is a little model with recursion, which does not benefit from a GPU). Currently, I assigned 1 CPU to each trial. The data is supplied to the model via a pytorch dataloader, which can be also parallelized setting num_workers (i.e., one trial can use num_workers to load the data). Without Ray Tune, I would set num_workers = <n_cpus>, but with Ray Tune already distributing trials across CPUs, I do not know what is the best value for num_workers.

Do you have any suggestions? Shall I set num_workers = 1 or would it be ok also to set num_workers = <n_cpus>?

2

Hi jack!
So you’re correct num_workers works entirely on how many CPUs you have available. You can definitely increase it to <n_cpus> but overall I would try to not exceed n_cpus to overload the existing resources.

There’s a few docs on this that I’m gonna link here that might be of interest to you!