# \[Core\] Ray.init() hanging

**URL:** <https://discuss.ray.io/t/core-ray-init-hanging/2138>\
**Category:** Ray Core\
**Created:** [May 11, 2021, 9:53pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138 "2021-05-11T21:53:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nathanlct](https://avatars.discourse-cdn.com/v4/letter/n/919ad9/32.png) [@nathanlct](https://discuss.ray.io/u/nathanlct)\
**Post date:** [May 11, 2021, 9:53pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138/1 "2021-05-11T21:53:18Z")

</div>

Hello,

I have a training script that runs well locally on my machine, but not always when I deploy it on an online machine (just on one node, not in cluster/autoscaler mode). After doing some debugging, it seems that I don’t get past the call to `ray.init()`, although I don’t get any explicit error either. Sometimes it works, sometimes it works after some (long) time, but often the initialisation never ends.

I’ve looked at the log of when it fails in `/tmp/ray/{expid}/logs` and I’m seeing some `Failed to register worker 135fa48518c1e9860c4f8ddae75d2dce6cee220df8b4882ed8e1b9f3 to Raylet. Invalid: Invalid: Unknown worker` errors. Also getting some warnings but I’m not sure which are relevant.

Here are all the log files (with `cat *`): [https://pastebin.com/raw/EKaHyiBx](https://pastebin.com/raw/EKaHyiBx). If someone has a clue of where I should look or what I could try to debug this, please let me know!

I’ve also tried specifying more info in the `ray.init` but with no success: `ray.init(include_dashboard=False, _temp_dir='/global/scratch/ray_logs', num_cpus=20, object_store_memory=500e6, _memory=500e6)` but without more success.

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [June 3, 2021, 2:39pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138/2 "2021-06-03T14:39:14Z")

</div>

Hey @nathanlct , sorry for the long wait. This is due to your question not being categorized. Could you make sure that for any new posts you add a category (e.g. “Ray Core” or “RLlib”) to your question? This helps us assign the right person to respond more quickly.

@Stephanie_Wang , could someone from the Ray Core team answer this one here?  
Thanks 🙂

---

<div class="post-metadata">

**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\
**Post date:** [November 24, 2021, 5:17pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138/3 "2021-11-24T17:17:31Z")

</div>

Sorry for the late reply. The reason is

> [2021-05-11 08:36:38,834 I 29410 29410] worker\_pool.cc:342: Some workers of the worker process(29490) have not registered to raylet within timeout.

So the timeout happens first and the worker is removed from raylet and later on when the worker tries to register, we cannot find it anymore.

You can increase the timeout by setting RAY\_worker\_register\_timeout\_seconds environment variable.

---

<div class="post-metadata">

**Author:** ![addisonklinke](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/addisonklinke/32/1906_2.png) [@addisonklinke](https://discuss.ray.io/u/addisonklinke)\
**Post date:** [December 21, 2021, 8:23pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138/4 "2021-12-21T20:23:36Z")

</div>

@jjyao This does not solve this issue for me, regardless of whether I set the environment variable via bash (`export RAY_worker_register_timeout_seconds=30`) or python (`os.environ['RAY_worker_register_timeout_seconds'] = '30'`).

My main script runs fine with `tune.run(resources_per_trial={'gpu': 1})`, but runs into the “Failed to register worker” error when using `gpus > 1`

---

<div class="post-metadata">

**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\
**Post date:** [December 21, 2021, 8:54pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138/5 "2021-12-21T20:54:07Z")

</div>

Hi @addisonklinke,

If that’s the case, could you file a github issue with a reproducible script? Thanks!

---

<div class="post-metadata">

**Author:** ![addisonklinke](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/addisonklinke/32/1906_2.png) [@addisonklinke](https://discuss.ray.io/u/addisonklinke)\
**Post date:** [December 21, 2021, 10:15pm UTC](https://discuss.ray.io/t/core-ray-init-hanging/2138/6 "2021-12-21T22:15:54Z")

</div>

@jjyao Certainly, I just filed a reproducible script from one of the official tutorials

[[Bug] Failed to register worker to Raylet for single node, multi-GPU #21226](https://github.com/ray-project/ray/issues/21226)
