# Ray Tune does not work properly with DDP PyTorch Lightning **URL:** \ **Category:** Ray Tune\ **Created:** [January 22, 2021, 3:14am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530 "2021-01-22T03:14:31Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![yongw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yongw/32/280_2.png) [@yongw](https://discuss.ray.io/u/yongw)\ **Post date:** [January 22, 2021, 3:14am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/1 "2021-01-22T03:14:31Z")
I found that Ray Tune does not work properly with DDP PyTorch Lightning. My specific situation is as follows. Ray 1.2.0.dev0, pytorch 1.7,pytorch lightning 1.1.1. I have one machine with 80 CPU cores and 2 GPUs. I want to use Ray Tune to carry out 1 trial, which requires 10 CPU cores and 2 GPUs.Using the DistributedDataParallel of PyTorch Lightning. I use “DistributedTrainableCreator” this way. DistributedTrainableCreator(train\_model\_with\_parameters, num\_workers= 1, num\_cpus\_per\_worker = 10,num\_gpus\_per\_worker = 2) But I got this warning. Soon after, the program ended unexpectedly. I observe that only GPU 0 is working and GPU 1 is not started. ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/10d9cfabdd8f744bad901c70546c16e7cc3f15e7.png) May I ask how this problem is caused and how to solve it? Thank you very much for your help! ---
**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\ **Post date:** [January 22, 2021, 9:33am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/2 "2021-01-22T09:33:33Z")
Hi @yongw, we recently received a number of messages concerning multi GPU training with PyTorch Lightning. We’re working on resolving this issue as soon as possible. Cc @amogkam who is working on this ---
**Author:** ![yongw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yongw/32/280_2.png) [@yongw](https://discuss.ray.io/u/yongw)\ **Post date:** [January 22, 2021, 10:36am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/3 "2021-01-22T10:36:58Z")
Thank you very much. ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [February 11, 2021, 7:57am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/4 "2021-02-11T07:57:57Z")
Hi @yongw, we just finished implementing a Ray backend for distributed Pytorch Lightning training here- [GitHub - ray-project/ray\_lightning\_accelerators: Pytorch Lightning Distributed Accelerators using Ray](https://github.com/ray-project/ray_lightning_accelerators). The package introduces 2 new Pytorch Lightning accelerators for quick and easy distributed training on Ray. It also integrates with Tune and should resolve your issue. Now you can use Tune to run multiple trials in parallel, and each trial can itself be distributed with any number of CPUs or GPUs. Please check it out, and let us know how it goes! ---
**Author:** ![yongw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yongw/32/280_2.png) [@yongw](https://discuss.ray.io/u/yongw)\ **Post date:** [February 18, 2021, 6:57am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/5 "2021-02-18T06:57:04Z")
Thanks for your reply. I’ve tried it and it works fine. I have one question about resource allocation. I have one machine with 80 CPU cores and 2 GPUs. should I have to set “num\_workers” =1. I don’t understand what “num\_workers” mean? ---
**Author:** ![yongw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yongw/32/280_2.png) [@yongw](https://discuss.ray.io/u/yongw)\ **Post date:** [February 18, 2021, 7:19am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/6 "2021-02-18T07:19:46Z")
Hi, @amogkam .For example , if I have one machine with 80 CPU cores and 8 GPUs. I want to use Ray Tune to carry out 3 trials, every trial needs 10 CPU cores and 2 GPUs, use DDP. how to set “cpu” ,“gpu”, “extra\_cpu”, “extra\_gpu” in ray.tune.run. how to set “num\_workers” , “cpus\_per\_worker” , “use\_gpu” in RayAccelerator. I’ve looked at this [ray\_ddp\_tune.py](https://github.com/ray-project/ray_lightning_accelerators/blob/17a35ba0a79e03cfef7be69f61236f5778b1629c/examples/ray_ddp_tune.py), but maybe it uses too few resources. Looking forward to your reply. ---
**Author:** ![yongw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yongw/32/280_2.png) [@yongw](https://discuss.ray.io/u/yongw)\ **Post date:** [February 18, 2021, 7:26am UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/7 "2021-02-18T07:26:54Z")
There is a little mistake in ReadMe. “cpus\_per\_worker” should change to “num\_cpus\_per\_worker” ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/42480175884be04c60c875174c887f0d7f350caf.png) ---
**Author:** ![Manoj\_Kumar\_Dobbali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manoj_kumar_dobbali/32/2172_2.png) [@Manoj\_Kumar\_Dobbali](https://discuss.ray.io/u/Manoj_Kumar_Dobbali)\ **Post date:** [March 15, 2022, 10:15pm UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/8 "2022-03-15T22:15:05Z")
> [@amogkam](#): > > It also integrates with Tune and should resolve your issue. Now you can use Tune to run multiple trials in parallel, and each trial can itself be distributed with any number of CPUs or GPUs. Hi, This is what I have been trying to achieve on a multi node multi GPU machine. It doesn’t seems to work as expected. Here([https://github.com/ray-project/ray\_lightning/blob/65f497a3c8bedb2f24bf04a5dbf0ea62b5bcb4d6/ray\_lightning/ray\_ddp.py#L84](https://github.com/ray-project/ray_lightning/blob/65f497a3c8bedb2f24bf04a5dbf0ea62b5bcb4d6/ray_lightning/ray_ddp.py#L84)) it says, specify GPUs in Pytorch Lightning Trainer to a value \> 0 and Here([https://github.com/ray-project/ray\_lightning/blob/65f497a3c8bedb2f24bf04a5dbf0ea62b5bcb4d6/ray\_lightning/ray\_ddp.py#L107](https://github.com/ray-project/ray_lightning/blob/65f497a3c8bedb2f24bf04a5dbf0ea62b5bcb4d6/ray_lightning/ray_ddp.py#L107)) in example it says NOT to specify resource in Trainer I have 5 worker nodes with each having 8 GPUs and 16 CPUS. Not sure how to allocated resources within RayPlugin & get\_tune\_resources function. ---
**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\ **Post date:** [March 17, 2022, 8:58pm UTC](https://discuss.ray.io/t/ray-tune-does-not-work-properly-with-ddp-pytorch-lightning/530/9 "2022-03-17T20:58:02Z")
Hi @Manoj_Kumar_Dobbali Can you share your script? I can help take a look!