# How to launch multi-node job with Ray Train? **URL:** https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540 **Category:** Ray Train **Created:** [December 29, 2021, 6:28pm UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540 "2021-12-29T18:28:26Z") **Posts on this page:** 10 **Page:** 1
### Author: ![Lacruche](https://avatars.discourse-cdn.com/v4/letter/l/ecccb3/32.png) [@Lacruche](https://discuss.ray.io/u/Lacruche) #### Post date: [December 29, 2021, 6:28pm UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/1 "2021-12-29T18:28:26Z")
Hi, I understand that the Ray Train trainer allows to launch a PyTorch single-host, multi-GPU job via this command: ```auto from ray.train import Trainer trainer = Trainer(backend="torch", num_workers=4) trainer.start() results = trainer.run(train_func_distributed) trainer.shutdown() ``` How to launch a PyTorch **multi-host** , multi-GPU job with Ray Train? shall we use MPI or something so that the trainer is launched on multiple nodes at the same time? ---
### Author: ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng) #### Post date: [December 29, 2021, 7:06pm UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/2 "2021-12-29T19:06:55Z")
Hi @Lacruche, This can be achieved by setting up your [Ray Cluster](https://docs.ray.io/en/master/cluster/index.html) with multiple nodes. Running the same script while connected to this cluster will then distribute the Ray Train workers across the hosts (and use DDP). With Ray Train, the same script that runs on 1 4-GPU node can be run on a cluster with 2 2-GPU nodes or a cluster with 4 1-GPU nodes! In general, MPI is not supported. By default, if you instantiate your `Trainer` with `use_gpu=True`, then the workers will use NCCL as the communication backend. This can be overridden to use Gloo by instantiating the `Trainer` with a [TorchConfig](https://docs.ray.io/en/master/train/api.html#torchconfig) that has `backend="gloo"`. ---
### Author: ![Lacruche](https://avatars.discourse-cdn.com/v4/letter/l/ecccb3/32.png) [@Lacruche](https://discuss.ray.io/u/Lacruche) #### Post date: [December 30, 2021, 1:52pm UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/3 "2021-12-30T13:52:22Z")
I’m using SageMaker-managed EC2, so I get a cluster of N EC2s already networked together. How can I create a Ray cluster on an already-running EC2 cluster? the Ray AWS doc seems to start from EC2 instance provisioning, which I don’t need would choosing a host as a coordinating server and running the below command ([from here](https://docs.ray.io/en/latest/cluster/cloud.html)) turn my N EC2s in a Ray cluster? `python coordinator_server.py --ips --port ` ---
### Author: ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng) #### Post date: [December 30, 2021, 6:22pm UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/4 "2021-12-30T18:22:13Z")
Ah I have yet to try that myself, but that seems reasonable. @Ameer_Haj_Ali can you confirm if using `coordinator_server.py` is a viable approach for creating a Ray cluster on an existing EC2 cluster? ---
### Author: ![hezhaozhao-git](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hezhaozhao-git/32/2264_2.png) [@hezhaozhao-git](https://discuss.ray.io/u/hezhaozhao-git) #### Post date: [March 11, 2022, 2:17am UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/5 "2022-03-11T02:17:59Z")
it is silent when i useing ray train on ray train multi-node multi-gpu, example i useing 2 node → 2gpu,total 4gpu2, is is silent ---
### Author: ![hezhaozhao-git](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hezhaozhao-git/32/2264_2.png) [@hezhaozhao-git](https://discuss.ray.io/u/hezhaozhao-git) #### Post date: [March 11, 2022, 2:19am UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/6 "2022-03-11T02:19:05Z")
| (BaseWorkerMixin pid=20102, ip=10.0.100.15) 2022-03-11 01:47:32,207 | INFO torch.py:247 – Wrapping provided model in DDP. | | --- | --- | | (BaseWorkerMixin pid=25342, ip=10.0.100.14) 2022-03-11 01:47:32,327 | INFO torch.py:247 – Wrapping provided model in DDP. | | (BaseWorkerMixin pid=25343, ip=10.0.100.14) 2022-03-11 01:47:32,369 | INFO torch.py:247 – Wrapping provided model in DDP. | | (BaseWorkerMixin pid=25344, ip=10.0.100.14) 2022-03-11 01:47:32,457 | INFO torch.py:247 – Wrapping provided model in DDP. | | (BaseWorkerMixin pid=25345, ip=10.0.100.14) 2022-03-11 01:47:32,455 | INFO torch.py:247 – Wrapping provided model in DDP. | this is log ---
### Author: ![hezhaozhao-git](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hezhaozhao-git/32/2264_2.png) [@hezhaozhao-git](https://discuss.ray.io/u/hezhaozhao-git) #### Post date: [March 11, 2022, 2:27am UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/7 "2022-03-11T02:27:08Z")
i run get result: “py-spy dump --native --pid 25344” Process 25344: ray::BaseWorkerMixin Python v3.8.12 (/home/gpu4/anaconda3/bin/python3.8) Thread 25344 (idle): “MainThread” epoll\_wait (libc-2.27.so) boost::asio::detail::epoll\_reactor::run (ray/\_raylet.so) boost::asio::detail::scheduler::do\_run\_one (ray/\_raylet.so) boost::asio::detail::scheduler::run (ray/\_raylet.so) boost::asio::io\_context::run (ray/\_raylet.so) ray::core::CoreWorker::RunTaskExecutionLoop (ray/\_raylet.so) ray::core::CoreWorkerProcessImpl::RunWorkerTaskExecutionLoop (ray/\_raylet.so) ray::core::CoreWorkerProcess::RunTaskExecutionLoop (ray/\_raylet.so) run\_task\_loop (ray/\_raylet.so) main\_loop (ray/worker.py:430) (ray/workers/default\_worker.py:224) Thread 25458 (idle): “ray\_import\_thread” do\_futex\_wait (libpthread-2.27.so) \_\_new\_sem\_wait\_slow (libpthread-2.27.so) PyThread\_acquire\_lock\_timed (python3.8) lock\_PyThread\_acquire\_lock (python3.8) wait (threading.py:306) \_wait\_once (grpc/\_common.py:106) wait (grpc/\_common.py:148) result (grpc/\_channel.py:733) \_poll\_locked (ray/\_private/gcs\_pubsub.py:270) poll (ray/\_private/gcs\_pubsub.py:409) \_run (ray/\_private/import\_thread.py:71) run (threading.py:870) \_bootstrap\_inner (threading.py:932) \_bootstrap (threading.py:890) clone (libc-2.27.so) Thread 2165 (idle): “Thread-41” epoll\_wait (libc-2.27.so) 0x7f6069f29bba (grpc/\_cython/cygrpc.cpython-38-x86\_64-linux-gnu.so) 0x7f6069fd142c (grpc/\_cython/cygrpc.cpython-38-x86\_64-linux-gnu.so) 0x7f6069cc5575 (grpc/\_cython/cygrpc.cpython-38-x86\_64-linux-gnu.so) 0x7f6069d28e47 (grpc/\_cython/cygrpc.cpython-38-x86\_64-linux-gnu.so) 0x7f6069d5a505 (grpc/\_cython/cygrpc.cpython-38-x86\_64-linux-gnu.so) channel\_spin (grpc/\_channel.py:1258) 0x7f6069c8caf9 (grpc/\_cython/cygrpc.cpython-38-x86\_64-linux-gnu.so) run (threading.py:870) \_bootstrap\_inner (threading.py:932) \_bootstrap (threading.py:890) clone (libc-2.27.so) Please tell me how can I change it so that he can train normally? thank you. ---
### Author: ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng) #### Post date: [March 11, 2022, 2:38am UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/8 "2022-03-11T02:38:59Z")
Can you create a new topic and post a reproduction script? ---
### Author: ![hezhaozhao-git](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hezhaozhao-git/32/2264_2.png) [@hezhaozhao-git](https://discuss.ray.io/u/hezhaozhao-git) #### Post date: [March 11, 2022, 2:59am UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/9 "2022-03-11T02:59:25Z")
yes,i can ,but I have submitted a [bug](https://github.com/ray-project/ray/issues/23060) in the github issue,you can go there to check and communicate with more people,thank you. ---
### Author: ![ynadell](https://avatars.discourse-cdn.com/v4/letter/y/4af34b/32.png) [@ynadell](https://discuss.ray.io/u/ynadell) #### Post date: [June 14, 2024, 9:16pm UTC](https://discuss.ray.io/t/how-to-launch-multi-node-job-with-ray-train/4540/10 "2024-06-14T21:16:19Z")
Is there a way to do this now? The links that you have are broken now.