# Worker nodes stuck in "waiting-for-ssh"

**URL:** <https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677>\
**Category:** Ray Clusters\
**Created:** [June 30, 2022, 7:36am UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677 "2022-06-30T07:36:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![dlozeve](https://avatars.discourse-cdn.com/v4/letter/d/df705f/32.png) [@dlozeve](https://discuss.ray.io/u/dlozeve)\
**Post date:** [June 30, 2022, 7:36am UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/1 "2022-06-30T07:36:28Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

Hello, I am trying to launch a Ray cluster on a local (self-hosted) set of servers. I have 1 head node and 2 worker nodes configured.

```auto
cluster_name: raytest
provider:
    type: local
    head_ip: frontal.cluster.lan.example.com
    worker_ips: [node1.cluster.lan.example.com, node2.cluster.lan.example.com]
auth:
    ssh_user: dimitri.lozeve
    ssh_private_key: ~/.ssh/id_ed25519

# [...] other options from example-full.yaml

```

When launching with `ray up -vvvvv --no-config-cache cluster-config.yaml`, the head node starts properly but the worker nodes get stuck in `waiting-for-ssh` node.

```auto
======== Autoscaler status: 2022-06-30 09:33:42.267203 ========
Node status
---------------------------------------------------------------
Healthy:

Pending:
 10.168.11.11: local.cluster.node, waiting-for-ssh
 10.168.11.12: local.cluster.node, waiting-for-ssh
 127.0.1.1: local.cluster.node, waiting-for-ssh
Recent failures:
 (no failures)

```

```auto
==> /tmp/ray/session_latest/logs/monitor.out <==
2022-06-30 09:33:42,397	VINFO command_runner.py:552 -- Running `uptime`
2022-06-30 09:33:42,397	VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/ray_bootstrap_key.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_4e3a8ef667/2283bd7c04/%C -o ControlPersist=10s -o ConnectTimeout=5s dimitri.lozeve@10.168.11.11 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
2022-06-30 09:33:42,402	VINFO command_runner.py:552 -- Running `uptime`
2022-06-30 09:33:42,402	VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/ray_bootstrap_key.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_4e3a8ef667/2283bd7c04/%C -o ControlPersist=10s -o ConnectTimeout=5s dimitri.lozeve@10.168.11.12 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`

==> /tmp/ray/session_latest/logs/monitor.err <==
dimitri.lozeve@10.168.11.11: Permission denied (publickey).

```

However, if I copy-paste the SSH command above on the head node, I can successfully login to the worker node (although there is a password prompt to unlock the private key).

---

<div class="post-metadata">

**Author:** ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)\
**Post date:** [June 30, 2022, 8:45pm UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/2 "2022-06-30T20:45:09Z")

</div>

> although there is a password prompt to unlock the private key

This sounds like the issue. Is there a way you can create a separate key without a password for the autoscaler?

---

<div class="post-metadata">

**Author:** ![dlozeve](https://avatars.discourse-cdn.com/v4/letter/d/df705f/32.png) [@dlozeve](https://discuss.ray.io/u/dlozeve)\
**Post date:** [July 1, 2022, 1:53pm UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/3 "2022-07-01T13:53:53Z")

</div>

Thank you! Indeed I think that was an issue. To sidestep the problem, I now launch the cluster from the head node directly, without specifying `ssh_private_key` (I believe this means that the autoscaler will generate a key for me).

Now the cluster seems to launch, but the workers get stuck at “setting-up”, and I get the following error:

```auto
2022-07-01 15:47:39,090 WARNING utils.py:1242 -- Unable to connect to GCS at 10.168.11.25:6380. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access.

```

The GCS is running on port 6380 and listening, and the ports are open.

From Python, `ray.nodes()` reports only the head node.

Do you know where this problem might come from?

---

<div class="post-metadata">

**Author:** ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)\
**Post date:** [July 1, 2022, 10:47pm UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/4 "2022-07-01T22:47:53Z")

</div>

Can you share some more details about the `initialization_commands` and `setup_commands` you have (if any).

Also what version of Ray is running on the worker nodes and is there a firewall between the head and worker nodes?

---

<div class="post-metadata">

**Author:** ![dlozeve](https://avatars.discourse-cdn.com/v4/letter/d/df705f/32.png) [@dlozeve](https://discuss.ray.io/u/dlozeve)\
**Post date:** [July 2, 2022, 8:47am UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/5 "2022-07-02T08:47:22Z")

</div>

Here are the relevant commands:

```nohighlight
setup_commands:
    - pip3 install -U "ray[default]"

head_start_ray_commands:
    - ray stop
    - ulimit -c unlimited && ray start --head --port=6380 --autoscaling-config=~/ray_bootstrap_config.yaml

worker_start_ray_commands:
    - ray stop
    - ray start --address=$RAY_HEAD_IP:6380

```

`initialization_commands`, `head_setup_commands`, and `worker_setup_commands` are all empty.

I am running Ray 1.13.0 on Python 3.9.2 (Debian 11). There is no firewall between the head and worker nodes, all the ports are open. (There is already a Redis service running on port 6379, so I configured Ray to use port 6380.)

---

<div class="post-metadata">

**Author:** ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)\
**Post date:** [July 5, 2022, 5:35pm UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/6 "2022-07-05T17:35:51Z")

</div>

I see, this should work… Do you mind ssh’ing into a worker node and sharing any erroring logs in `/tmp/ray/session_latest` on the worker nodes? To access a worker node you can do` ssh -i ~/bootstrap_key.pem ubuntu@<worker-ip>` on the head node.

---

<div class="post-metadata">

**Author:** ![dlozeve](https://avatars.discourse-cdn.com/v4/letter/d/df705f/32.png) [@dlozeve](https://discuss.ray.io/u/dlozeve)\
**Post date:** [July 6, 2022, 6:55am UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/7 "2022-07-06T06:55:20Z")

</div>

There is no `/tmp/ray` directory on any of the worker nodes…

In `ray monitor cluster-config.yaml`, I still have the “Unable to connect to GCS” error, plus this:

```auto
==> /tmp/ray/session_latest/logs/monitor.out <==                                                                                                                                                                     
2022-07-06 08:51:48,200 WARNING utils.py:1242 -- Unable to connect to GCS at 127.0.1.1:6380. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall
 setting preventing access.                                                                                                                                                                                          
2022-07-06 08:51:48,335 WARNING utils.py:1242 -- Unable to connect to GCS at 127.0.1.1:6380. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall
 setting preventing access.                                                                                                                                                                                          
Traceback (most recent call last):                                                                                                                                                                                   
  File "/nfs/home/dimitri.lozeve/.local/bin/ray", line 8, in <module>                                                                                                                                                
    sys.exit(main())                                                                                                                                                                                                 
  File "/nfs/home/dimitri.lozeve/.local/lib/python3.9/site-packages/ray/scripts/scripts.py", line 2339, in main
    return cli()
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/nfs/home/dimitri.lozeve/.local/lib/python3.9/site-packages/ray/autoscaler/_private/cli_logger.py", line 852, in wrapper
    return f(*args, **kwargs)
  File "/nfs/home/dimitri.lozeve/.local/lib/python3.9/site-packages/ray/scripts/scripts.py", line 872, in start
    node = ray.node.Node(
  File "/nfs/home/dimitri.lozeve/.local/lib/python3.9/site-packages/ray/node.py", line 185, in __init__
    session_name = ray._private.utils.internal_kv_get_with_retry(
  File "/nfs/home/dimitri.lozeve/.local/lib/python3.9/site-packages/ray/_private/utils.py", line 1258, in internal_kv_get_with_retry
    raise RuntimeError(
RuntimeError: Could not read 'session_name' from GCS. Did GCS start successfully?
2022-07-06 08:51:50,272 WARNING utils.py:1242 -- Unable to connect to GCS at 127.0.1.1:6380. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall
 setting preventing access.

```

In case it’s useful, I can see these `.err` log files on the head node:

```auto
$ cat ray_client_server.err 
2022-07-06 08:49:15,226 INFO server.py:843 -- Starting Ray Client server on 0.0.0.0:10001

```

```auto
$ cat monitor.err
Shared connection to 10.168.11.12 closed.
Shared connection to 10.168.11.11 closed.
Shared connection to 10.168.11.25 closed.
Shared connection to 10.168.11.12 closed.
Shared connection to 10.168.11.11 closed.
Shared connection to 10.168.11.25 closed.
Shared connection to 10.168.11.12 closed.
Shared connection to 10.168.11.11 closed.
Shared connection to 10.168.11.25 closed.

```

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![dlozeve](https://avatars.discourse-cdn.com/v4/letter/d/df705f/32.png) [@dlozeve](https://discuss.ray.io/u/dlozeve)\
**Post date:** [July 6, 2022, 7:42am UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/8 "2022-07-06T07:42:06Z")

</div>

What I find surprising, is that I can launch a Ray cluster manually without issue.

On the head node:

```bash
ray start --head --port=6380

```

On the worker(s):

```bash
ray start --address='10.168.11.10:6380'

```

And everything works fine. I would just like to use `ray up` to automate this process, and use the YAML file as a kind of “infrastructure as code”. I’m not sure what the autoscaler does differently?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [July 6, 2022, 5:11pm UTC](https://discuss.ray.io/t/worker-nodes-stuck-in-waiting-for-ssh/6677/9 "2022-07-06T17:11:43Z")

</div>

The autoscaler provides no benefit here besides simplifying the startup process. It looks like that’s not working in your case, unfortunately.  
I’d recommend writing a simple script to automate the manual process you’ve figured out.