# Horovod Trainer hangs

**URL:** <https://discuss.ray.io/t/horovod-trainer-hangs/9942>\
**Category:** Ray Train\
**Created:** [March 29, 2023, 12:25am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942 "2023-03-29T00:25:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mizhazha](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mizhazha/32/3047_2.png) [@mizhazha](https://discuss.ray.io/u/mizhazha)\
**Post date:** [March 29, 2023, 12:25am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942/1 "2023-03-29T00:25:56Z")

</div>

- High: It blocks me to complete my task.

I use Horovod trainer. The train job hangs after many iterations without failing. The horovod timeout should be whatever ray default.  
I expect the job to fail with exception thrown. Now the job status is still running but the GPU is not been utilized and the training logs doesn’t update.

Can someone suggest any debugging tips?

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [March 29, 2023, 12:53am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942/2 "2023-03-29T00:53:50Z")

</div>

Hey, one suggestion is to use the Ray Dashboard to [find the stack trace](https://docs.ray.io/en/master/ray-observability/monitoring-debugging/profiling.html#python-cpu-profiling-in-the-dashboard) of the training worker processes, to identify where the job is hanging.

---

<div class="post-metadata">

**Author:** ![mizhazha](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mizhazha/32/3047_2.png) [@mizhazha](https://discuss.ray.io/u/mizhazha)\
**Post date:** [March 29, 2023, 3:01am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942/3 "2023-03-29T03:01:25Z")

</div>

Thanks Matthew for taking a look. The stack trace is as follow:

```auto
Process 30141: ray::RayTrainWorker._RayTrainWorker__execute
Python v3.10.2 (/export/apps/python/3.10.2/bin/python3.10)

Thread 30141 (idle): "MainThread"
    wait (threading.py:324)
    get (queue.py:180)
    get_next (ray/train/_internal/session.py:175)
    get_next (ray/train/_internal/backend_executor.py:425)
    __execute (ray/train/_internal/worker_group.py:28)
    _resume_span (ray/util/tracing/tracing_helper.py:466)
    actor_method_executor (ray/_private/function_manager.py:674)
    main_loop (ray/_private/worker.py:763)
    <module> (ray/_private/workers/default_worker.py:233)
Thread 30233 (idle): "ray_import_thread"
    wait (threading.py:324)
    _wait_once (grpc/_common.py:112)
    wait (grpc/_common.py:157)
    result (grpc/_channel.py:733)
    _poll_locked (ray/_private/gcs_pubsub.py:255)
    poll (ray/_private/gcs_pubsub.py:391)
    _run (ray/_private/import_thread.py:69)
    run (threading.py:946)
    _bootstrap_inner (threading.py:1009)
    _bootstrap (threading.py:966)
Thread 30349 (idle): "Thread-11 (trainable_fn)"
    quick_execute (tensorflow/python/eager/execute.py:52)
    call (tensorflow/python/eager/polymorphic_function/monomorphic_function.py:378)
    _call_flat (tensorflow/python/eager/polymorphic_function/monomorphic_function.py:1745)
    __call__ (tensorflow/python/eager/polymorphic_function/tracing_compiler.py:134)
    _call (tensorflow/python/eager/polymorphic_function/polymorphic_function.py:912)
    __call__ (tensorflow/python/eager/polymorphic_function/polymorphic_function.py:880)
    error_handler (tensorflow/python/util/traceback_utils.py:150)
    fit (keras/engine/training.py:1650)
    error_handler (keras/utils/traceback_utils.py:65)
    train (trainer.py:270)
    main (libenchmarkingpipeline/run_finetune_and_eval_pipeline.py:171)
    trainable_fn (utils/convert_to_ray_trainable.py:65)
    discard_return_wrapper (ray/train/_internal/utils.py:129)
    train_fn (ray/train/_internal/utils.py:149)
    run (ray/air/_internal/util.py:88)
    _bootstrap_inner (threading.py:1009)
    _bootstrap (threading.py:966)
Thread 32010 (idle): "Thread-61 (_run)"
    channel_spin (grpc/_channel.py:1258)
    run (contextvars/ __init__.py:38)
    run (threading.py:946)
    _bootstrap_inner (threading.py:1009)
    _bootstrap (threading.py:966)

```

I am not sure how to interpret the stack trace here. Any insight?

---

<div class="post-metadata">

**Author:** ![mizhazha](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mizhazha/32/3047_2.png) [@mizhazha](https://discuss.ray.io/u/mizhazha)\
**Post date:** [March 29, 2023, 3:32pm UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942/4 "2023-03-29T15:32:54Z")

</div>

I have found the root cause. It is because the training data was not sharded properly so training finished earlier for some workers than the other. So hypothetically horovod will timeout.

I expect Ray to let the job fail instead of hanging there for hours. Do I need to set anything to get the expected behavior?

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [March 30, 2023, 2:02am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942/5 "2023-03-30T02:02:29Z")

</div>

Oh nice, glad you were able to find the root cause!

Hmm do you know if Horovod (or TensorFlow) would actually timeout in this scenario? My understanding is that hanging may still occur while waiting to synchronize the gradients.

---

<div class="post-metadata">

**Author:** ![localh](https://avatars.discourse-cdn.com/v4/letter/l/87869e/32.png) [@localh](https://discuss.ray.io/u/localh)\
**Post date:** [November 3, 2023, 5:14am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942/6 "2023-11-03T05:14:07Z")

</div>

How did you know / could check that the data was not sharded properly? What did you do to fix it? I possibly have the same issue.