# Not able to ssh into head node during ray up

**URL:** <https://discuss.ray.io/t/not-able-to-ssh-into-head-node-during-ray-up/6457>\
**Category:** Ray Clusters\
**Created:** [June 9, 2022, 11:00pm UTC](https://discuss.ray.io/t/not-able-to-ssh-into-head-node-during-ray-up/6457 "2022-06-09T23:00:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dorekhov1](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dorekhov1/32/2697_2.png) [@dorekhov1](https://discuss.ray.io/u/dorekhov1)\
**Post date:** [June 9, 2022, 11:00pm UTC](https://discuss.ray.io/t/not-able-to-ssh-into-head-node-during-ray-up/6457/1 "2022-06-09T23:00:05Z")

</div>

This is the `ray_conf.yaml` I am using:

```auto
cluster_name: default

provider:
    type: local
    head_ip: 109.248.175.190
    worker_ips: [185.244.175.188]

auth:
    ssh_user: ubuntu
    ssh_private_key: ~/.ssh/ed25519
min_workers: 1
max_workers: 1

file_mounts: {
  "~/environment.yml": "/home/daniil/Lania/dragonshore/environment.yml"
}

setup_commands: 
  - conda env create -f ~/environment.yml || conda env update -f ~/environment.yml

head_start_ray_commands:
  - conda activate ray && mlflow server --backend-store-uri mlflow_data --host 0.0.0.0 -p 8889 &> mlflow.log &
  - conda activate ray && ray stop
  - conda activate ray && ulimit -c unlimited && ray start --head --port=6379 --autoscaling-config=~/ray_bootstrap_config.yaml

worker_start_ray_commands:
  - conda activate ray && ray stop
  - conda activate ray && ray start --address=$RAY_HEAD_IP:6379

```

after running `ray up ray_conf.yaml --no-config-cache -v` I get the following error:

```auto
Cluster: default

File Mount: (~/environment.yml:/home/daniil/ray_test/environment.yml) refers to a file.
 To ensure this mount updates properly, please use a directory.
Checking Local environment settings
2022-06-09 18:54:34,504 INFO node_provider.py:49 -- ClusterState: Loaded cluster state: ['185.244.175.188', '109.248.175.190']
Updating cluster configuration and running full setup.
Cluster Ray runtime will be restarted. Confirm [y/N]: y

Usage stats collection is enabled. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.

<1/1> Setting up head node
  Prepared bootstrap config
2022-06-09 18:54:35,534 INFO node_provider.py:110 -- ClusterState: Writing cluster state: ['185.244.175.188', '109.248.175.190']
  New status: waiting-for-ssh
  [1/7] Waiting for SSH to become available
    Running `uptime` as a test.
    Fetched IP: 109.248.175.190
    Running `uptime`
Shared connection to 109.248.175.190 closed.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    Running `uptime`
Shared connection to 109.248.175.190 closed.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    Running `uptime`
Shared connection to 109.248.175.190 closed.
    SSH still not available (SSH command failed.), retrying in 5 seconds.

```

And this keeps going forever.

python: 3.9.12  
ray: 1.13.0 (same thing for 1.12.1)

Interestingly, if I use `--use-normal-shells`, then it is able to connect with SSH just fine (but I then get other issues with it not going through my `.zshrc` file, so running conda breaks)

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [June 10, 2022, 12:06am UTC](https://discuss.ray.io/t/not-able-to-ssh-into-head-node-during-ray-up/6457/2 "2022-06-10T00:06:19Z")

</div>

Would you mind re-running with more v’s and sharing the output?  
`ray up ray_conf.yaml --no-config-cache -vvvvvvvvv`

---

<div class="post-metadata">

**Author:** ![cade](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cade/32/3837_2.png) [@cade](https://discuss.ray.io/u/cade)\
**Post date:** [June 15, 2022, 9:10pm UTC](https://discuss.ray.io/t/not-able-to-ssh-into-head-node-during-ray-up/6457/3 "2022-06-15T21:10:18Z")

</div>

Hi @dorekhov1, could you provide more verbose output so we can diagnose this? Were you able to work around the issue?

---

<div class="post-metadata">

**Author:** ![dorekhov1](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dorekhov1/32/2697_2.png) [@dorekhov1](https://discuss.ray.io/u/dorekhov1)\
**Post date:** [June 17, 2022, 8:13pm UTC](https://discuss.ray.io/t/not-able-to-ssh-into-head-node-during-ray-up/6457/4 "2022-06-17T20:13:27Z")

</div>

Yeah, I fixed it. The problem was that I was expecting conda to install ray on the cluster, but it turns out that the cluster needs ray installed before it can run anything.