# torch.nn.DataParallel with tune.run()

**URL:** <https://discuss.ray.io/t/torch-nn-dataparallel-with-tune-run/6644>\
**Category:** Ray Tune\
**Created:** [June 27, 2022, 6:13pm UTC](https://discuss.ray.io/t/torch-nn-dataparallel-with-tune-run/6644 "2022-06-27T18:13:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![flpgrz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/flpgrz/32/2805_2.png) [@flpgrz](https://discuss.ray.io/u/flpgrz)\
**Post date:** [June 27, 2022, 6:13pm UTC](https://discuss.ray.io/t/torch-nn-dataparallel-with-tune-run/6644/1 "2022-06-27T18:13:54Z")

</div>

Hello,

on a server I have 5 GPUs.  
Imagine a single torch model does not fit on a single GPU, hence I have to use `torch.nn.DataParallel`. This splits the batches across all available GPUs.

Now, I want to do hyperparameter optimization using Ray (`tune.run()`). Even though I wrap my torch model in `torch.nn.DataParallel`, it seems that Ray ignores that, and still tries to use 1 GPU per model. This throws a memory error.

How can I distribute one single trial on multiple GPUs?  
I tried `resources_per_trial={"gpu": 5}` , but no success.

Thanks.

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [June 28, 2022, 5:12am UTC](https://discuss.ray.io/t/torch-nn-dataparallel-with-tune-run/6644/2 "2022-06-28T05:12:29Z")

</div>

> Imagine a single torch model does not fit on a single GPU

By this do you mean the model does not fit on the GPU or a single batch does not fit on the GPU?

If you are indeed looking for data parallelism, I’d recommend checking out [Ray Train](https://docs.ray.io/en/releases-1.13.0/train/train.html)! Also, in general it is recommended to use [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) instead of `DataParallel`.

If your _model_ does not fit on the GPU, you may need to use [Pipeline Parallelism](https://pytorch.org/docs/stable/pipeline.html).