# Ray Train with Horovod does not use all GPUs on the node **URL:** \ **Category:** Ray Train\ **Created:** [April 7, 2022, 6:57am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717 "2022-04-07T06:57:51Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\ **Post date:** [April 7, 2022, 6:57am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/1 "2022-04-07T06:57:51Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. I’m just having a great time using Ray Train. It’s interface is so simple and yet very powerful! However, when I use Tensorflow Keras + Horovod for distributed multiworker training, my script doesn’t use all the GPUs on the node. It only uses GPU0 on all the workers as shown below. ```python trainer = Trainer(backend="horovod", num_workers=2, resources_per_worker={"GPU": 2, "CPU": 4}, use_gpu=True, max_retries=0) results = trainer.run(train_fn, config, callbacks=callbacks) ``` Without Train, it works as expected when I launch the training like below, ```bash ./horovodrun -np 4 -H ip1:2,ip2:2 python train_deep_model.py ``` What am I missing? How can I debug it? ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [April 7, 2022, 4:40pm UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/2 "2022-04-07T16:40:09Z")
Hey @Nitin_Pasumarthy thanks for posting the question! Can you provide more details on the cluster configuration and also what exactly you are seeing? It seems like you have 2 nodes with 2 GPUs each, is that right? And only the first GPU is being used on each node? If the above is correct, then this is expected based on how the Trainer is being instantiated. Unlike with horovodrun, with Ray Train, there’s no concept of separate hosts or nodes. You simply have to specify the number of workers (i.e. the number of training processes). So if you have a total of 4 GPUs in your cluster, regardless of whether these GPUs are on a single node or multiple nodes, all you need to do is just this `Trainer(backend="horovod", num_workers=4, use_gpu=True)` This would be the equivalent to the `horovodrun` call that you have. Currently, you are only creating 2 training workers, so if your `train_fn` only uses 1 GPU, then a total of 2 out of the 4 GPUs will be used. ---
**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\ **Post date:** [April 8, 2022, 1:58am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/3 "2022-04-08T01:58:12Z")
Thanks for taking a look, @amogkam. Yes, you understood my setup correctly - I have 2 nodes with 2 GPUs each where only 1 GPU from each node is being utilized. - `Trainer(backend="horovod", num_workers=4, use_gpu=True)` throws the below error, ```auto repr=) File "/site-packages/ray/train/backend.py", line 176, in start self._backend.on_start(self.worker_group, self._backend_config) File "/site-packages/ray/train/horovod.py", line 145, in on_start node_workers=node_workers) File "/site-packages/horovod/ray/utils.py", line 72, in detect_nics settings) File "/site-packages/horovod/ray/driver_service.py", line 47, in _driver_fn raise ValueError(f"Number of node actors ({len(node_actors)}) " ValueError: Number of node actors (4) must match num_hosts (2). ``` - `Trainer(backend="horovod", num_workers=3, use_gpu=True)` fails with the same error above. - `Trainer(backend="horovod", num_workers=2, use_gpu=True)` works and schedules the job on the same node, which is good to be honest. This is unlike the setup with `resources_per_worker={"GPU": 2, "CPU": 4}` arg, where the job gets scheduled on the different nodes, which is expected. But still multi worker training is unsolved with horovod. ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [April 8, 2022, 3:48am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/4 "2022-04-08T03:48:13Z")
Ah @Nitin_Pasumarthy can you use ray 1.12.0.rc1 to fix the ValueError that you saw? There was a bug with horovod training with older versions of ray. ---
**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\ **Post date:** [April 8, 2022, 4:09am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/5 "2022-04-08T04:09:19Z")
I was partially guessing this was a bug and thought I should mention the version of Ray I was using. Thank you! I’ll give it a try. When is the next stable release of Ray? ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [April 8, 2022, 4:37am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/6 "2022-04-08T04:37:13Z")
The official 1.12 release should be in the next few days! ---
**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\ **Post date:** [April 16, 2022, 7:12am UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/7 "2022-04-16T07:12:14Z")
Hey @amogkam, I checked with ray==1.12 and still unable to use all my 4 GPUs (2x2 nodes). It throws the same error. Would it be easier if we can debug this live, together? This is really blocking my progress ☹ I see some more minor errors / issues which I posted as separate threads or +1s to existing threads. ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [April 18, 2022, 4:33pm UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/8 "2022-04-18T16:33:29Z")
Hey @Nitin_Pasumarthy sorry I mispoke- this fix was not included in the 1.12.0 release yet. It should be fixed in master! Regarding the other issue you are seeing, I will respond on the other thread! ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [April 18, 2022, 4:35pm UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/9 "2022-04-18T16:35:02Z")
And yes would be happy to schedule some time to pair debug this! Feel free to message me on the Ray slack! ---
**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\ **Post date:** [June 3, 2022, 9:46pm UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/10 "2022-06-03T21:46:44Z")
@amogkam was this merged in 1.12.1 ? ---
**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\ **Post date:** [June 6, 2022, 9:24pm UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/11 "2022-06-06T21:24:16Z")
Hey @Nitin_Pasumarthy, no this will be included in Ray 1.13 release which will happen in the next few days. Were you able to try Ray master? ---
**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\ **Post date:** [June 8, 2022, 9:05pm UTC](https://discuss.ray.io/t/ray-train-with-horovod-does-not-use-all-gpus-on-the-node/5717/12 "2022-06-08T21:05:19Z")
Got it. Not yet, @amogkam . I’m working to overcome this limitation - [Train with tune doesnt set the right logdir - #7 by matthewdeng](https://discuss.ray.io/t/train-with-tune-doesnt-set-the-right-logdir/5471/7) which is a blocker for my work.