# Ray Tune trials getting stuck in a deadlock

**URL:** <https://discuss.ray.io/t/ray-tune-trials-getting-stuck-in-a-deadlock/12067>\
**Category:** Ray Tune\
**Created:** [September 7, 2023, 8:45am UTC](https://discuss.ray.io/t/ray-tune-trials-getting-stuck-in-a-deadlock/12067 "2023-09-07T08:45:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karapo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/karapo/32/5014_2.png) [@Karapo](https://discuss.ray.io/u/Karapo)\
**Post date:** [September 7, 2023, 8:45am UTC](https://discuss.ray.io/t/ray-tune-trials-getting-stuck-in-a-deadlock/12067/1 "2023-09-07T08:45:22Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

I’m running several ML experiments with Python/PyTorch on a shared server with several GPUs and CPUs. The trials start normally and proceed for a while until some of them just stop. E.g. the pink line has done only 2 steps.

 ![Screenshot from 2023-09-07 08-48-59](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/8/8febe5dc4c465736958e3ee9116104acd8b372f9.png)

Looking at the dashboard, it seems that the trials get stuck in the training epoch. By looking at the tasks for stuck trial, I usually see one .train running for minutes/hours even though a single epoch should just take ~30/40s

Digging deeper, I decided to place [tqdm](https://tqdm.github.io/) on the batch iterator and it looks like the trial gets stuck at the beginning of the epoch. E.g. stderr file from one stuck task:

```auto
16%|█▌ | 8/50 [04:54<26:18, 37.59s/it] # epoch counter
0%| | 0/887 [00:00<?, ?it/s]e[A # batch counter did not start

```

Following this, I tried to adjust the num\_workers in the Pytorch dataloader, and **setting num\_workers=0** solves the issue. However, this comes at a cost that the runs will take ages to finish.

I still cannot explain this to myself and I’m not sure what causes this, especially because:

- Running the same type of experiments on my local machine with ray tune and num\_workers\>0 works fine
- Running single trials without ray tune on the server with num\_workers \> 0 also works fine

**Does anybody know what is happening and why having multiple workers in the dataloader leads to ray tune to get stuck (on some machine)?**

Other additional info:

- No major python package differences (from what I can see) between the server and my local machine
- Even if I run ~4 trials on a single GPU with num\_workers\>0 it works fine.
- Packages versions:

```auto
ray: 2.6.3
python: 3.8.13
pytorch: 1.10.2 
CUDA: 11.1.74 

```

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [September 13, 2023, 12:54am UTC](https://discuss.ray.io/t/ray-tune-trials-getting-stuck-in-a-deadlock/12067/2 "2023-09-13T00:54:39Z")

</div>

Hey are you still running into this problem? One suggestion is to use the Ray Dashboard to [check the stacktrace](https://docs.ray.io/en/master/ray-observability/user-guides/debug-apps/debug-hangs.html), which might indicate where the processes are hanging.