# How to analysis or debug the connecting procedure

**URL:** <https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291>\
**Category:** Uncategorized\
**Created:** [March 17, 2021, 8:17am UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291 "2021-03-17T08:17:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![crystalww](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/crystalww/32/39_2.png) [@crystalww](https://discuss.ray.io/u/crystalww)\
**Post date:** [March 17, 2021, 8:17am UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/1 "2021-03-17T08:17:10Z")

</div>

As mentioned before ([[ray1.0.0] stuck when connecting to existing ray cluster](https://discuss.ray.io/t/ray1-0-0-stuck-when-connecting-to-existing-ray-cluster/225)), I met the same problem again, Worker Node can’t connect to Head Node, but all ports needed is open (tested by telnet).

```auto
>>> ray.init(address="auto")
2021-03-17 16:09:09,013 INFO worker.py:634 -- Connecting to existing Ray cluster at address: 192.168.250.10:6379

```

It stucking with nothing output.

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [March 17, 2021, 11:32pm UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/2 "2021-03-17T23:32:18Z")

</div>

What version of Ray are you using + What’s your operating systems?

---

<div class="post-metadata">

**Author:** ![crystalww](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/crystalww/32/39_2.png) [@crystalww](https://discuss.ray.io/u/crystalww)\
**Post date:** [March 18, 2021, 7:09am UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/3 "2021-03-18T07:09:42Z")

</div>

- CentOS Linux release 7.8.2003
- Ray(1.0.0)
- Python3.6.9

Seems stucking in **ray.\_raylet.CoreWorker**

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/bd73295dacaa83d013f32d43303d32bc1f05ef1b.png)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/f9f10f507321cce586fb75b2863b80b658d4d3fc.png)

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [March 18, 2021, 7:22pm UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/4 "2021-03-18T19:22:56Z")

</div>

Do you use VPN? Also, if you just do this;

```auto
ray start --head
# start a script that runs ray.init(address='auto')

```

Does it still hang?

---

<div class="post-metadata">

**Author:** ![crystalww](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/crystalww/32/39_2.png) [@crystalww](https://discuss.ray.io/u/crystalww)\
**Post date:** [March 19, 2021, 5:59am UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/5 "2021-03-19T05:59:08Z")

</div>

I’m using `docker run --net host ...`, no VPN.  
And as did as you say, but sometimes it hang in the head node also.

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [March 19, 2021, 10:21pm UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/6 "2021-03-19T22:21:09Z")

</div>

Hmm usually this kind of hanging has lots of different reasons. If you use the more recent version of ray, it still happens? If not, I’d like to probably ask for a pair debugging to help you unresolving it.

---

<div class="post-metadata">

**Author:** ![crystalww](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/crystalww/32/39_2.png) [@crystalww](https://discuss.ray.io/u/crystalww)\
**Post date:** [March 23, 2021, 9:49am UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/7 "2021-03-23T09:49:49Z")

</div>

I tried Ray 1.2.0, It can connect to head, but all tasks executed in the same node which run the python script (though all other nodes are idle), and there is no much detail logs. The raylet.out shows:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/14997f8fd214d539d72dd00f140c6c8c0ef023c5.png)

---

<div class="post-metadata">

**Author:** ![crystalww](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/crystalww/32/39_2.png) [@crystalww](https://discuss.ray.io/u/crystalww)\
**Post date:** [March 23, 2021, 12:51pm UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/8 "2021-03-23T12:51:05Z")

</div>

Actually, Ray 1.2 has more strange problems compare to Ray 1.0 when I run the same project code.  
In Ray 1.2.0, The driver shows：

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/22cd7daf46a530b9aaf747857f262113adf5c1b1.png)  
**debug\_state.txt**  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/76ce84f9bd7df64321fd03751924f2c7573649c0.png)

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [March 23, 2021, 5:26pm UTC](https://discuss.ray.io/t/how-to-analysis-or-debug-the-connecting-procedure/1291/9 "2021-03-23T17:26:00Z")

</div>

1. Tasks executed on the same node are expected. Tasks are started to be scheduled on a remote node once the local node is saturated. If you set your head node cpu == 0, are those tasks scheduled?
2. The messages can happen for various reasons. Are your actors created eventually, or is it hanging?

Also, if you are in public slack, I’d love to take a look at your issue in a video chat.