# What is the right way of using Ray tune with Pytorch DDP

**URL:** <https://discuss.ray.io/t/what-is-the-right-way-of-using-ray-tune-with-pytorch-ddp/13793>\
**Category:** Ray Tune\
**Created:** [February 21, 2024, 8:05pm UTC](https://discuss.ray.io/t/what-is-the-right-way-of-using-ray-tune-with-pytorch-ddp/13793 "2024-02-21T20:05:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![veydan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/veydan/32/5757_2.png) [@veydan](https://discuss.ray.io/u/veydan)\
**Post date:** [February 21, 2024, 8:05pm UTC](https://discuss.ray.io/t/what-is-the-right-way-of-using-ray-tune-with-pytorch-ddp/13793/1 "2024-02-21T20:05:41Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

I searched around and didn’t find a good answer of using Ray tune alone with Pytorch DDP model.

I am able to use the nn.DataParallel to wrap the model and run on single node. For DDP, usually we use mp.spawn() or torchrun to launch multiple processes each with a different “rank”, where should I put the tune.run(), inside each process or outside? In general, I am not sure how the Ray tune processes work with the multiple processes launched by PyTorch. Has anyone successfully run hyperparameter tuning with tune + ddp? Thanks.

I try to use the prepare\_model() in a train function to create DDP model, but the world\_size is None,

```auto
  File "python3.11/site-packages/ray/train/torch/train_loop_utils.py", line 328, in prepare_model
    if parallel_strategy and world_size > 1:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

```

My code flow looks like the follows,

```auto
def train_func(config):
    .... ...
    prepare_model(my_model)
    ... ...

tune.run(
        tune.with_parameters(
            train_func, ... ...),
        resources_per_trial={"cpu": 2, "gpu": 4},
        config=param_space,
        num_samples=1,
    )

```

---

<div class="post-metadata">

**Author:** ![yunxuanx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yunxuanx/32/4354_2.png) [@yunxuanx](https://discuss.ray.io/u/yunxuanx)\
**Post date:** [February 23, 2024, 10:33pm UTC](https://discuss.ray.io/t/what-is-the-right-way-of-using-ray-tune-with-pytorch-ddp/13793/2 "2024-02-23T22:33:56Z")

</div>

Hi @veydan , the best way is to use `TorchTrainer` + `Tuner`. You can refer to this example for more details: [Using PyTorch Lightning with Tune — Ray 3.0.0.dev0](https://docs.ray.io/en/master/tune/examples/tune-pytorch-lightning.html#tune-pytorch-lightning-ref)