As requested, here is a ray up -vvv followed by ssh -vvv on the command that failed. Thankfully ssh also looked like it failed this time, so hopefully it will help with reproduction. It looks like ssh “exited” but was still producing output to the terminal, as can be seen from the terminal prompt a few lines above the end ((env) chris_chiasson@penguin:~/te$ debug3: send packet: type 80). I waited a while and then copied all the output. In the copy/paste, I replaced the project name and my local directory name to protect my privacy.
(env) chris_chiasson@penguin:~/te$ ray up -vvv gpu-cluster-full.yaml
Cluster: default
Loaded cached provider configuration from /tmp/ray-config-9516e8899f656f94897811887e256109bf9f4f05
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
No head node found. Launching a new cluster. Confirm [y/N]: y
Acquiring an up-to-date head node
2021-09-06 18:29:30,309 INFO node.py:295 -- wait_for_compute_zone_operation: Waiting for operation operation-1630970969562-5cb5c05479ceb-1feec835-790ea88d to finish...
2021-09-06 18:30:01,931 INFO node.py:307 -- wait_for_compute_zone_operation: Operation operation-1630970969562-5cb5c05479ceb-1feec835-790ea88d finished.
Launched a new head node
Fetching the new head node
<1/1> Setting up head node
Prepared bootstrap config
2021-09-06 18:30:02,981 INFO node.py:295 -- wait_for_compute_zone_operation: Waiting for operation operation-1630971003149-5cb5c07481e4b-d03f14be-23cf3045 to finish...
2021-09-06 18:30:08,419 INFO node.py:307 -- wait_for_compute_zone_operation: Operation operation-1630971003149-5cb5c07481e4b-d03f14be-23cf3045 finished.
New status: waiting-for-ssh
[1/7] Waiting for SSH to become available
Running `uptime` as a test.
Fetched IP: 34.69.71.176
Running `uptime`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
ssh: connect to host 34.69.71.176 port 22: Connection timed out
SSH still not available (SSH command failed.), retrying in 5 seconds.
Running `uptime`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
Warning: Permanently added '34.69.71.176' (ECDSA) to the list of known hosts.
ubuntu@34.69.71.176: Permission denied (publickey).
SSH still not available (SSH command failed.), retrying in 5 seconds.
Running `uptime`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
Warning: Permanently added '34.69.71.176' (ECDSA) to the list of known hosts.
23:30:34 up 0 min, 1 user, load average: 1.49, 0.37, 0.12
Shared connection to 34.69.71.176 closed.
Success.
Updating cluster configuration. [hash=a704061c3833450f2e7b7580855544aaccc71da2]
2021-09-06 18:30:34,532 INFO node.py:295 -- wait_for_compute_zone_operation: Waiting for operation operation-1630971034721-5cb5c0929db4e-b76aff0a-79749d99 to finish...
2021-09-06 18:30:39,931 INFO node.py:307 -- wait_for_compute_zone_operation: Operation operation-1630971034721-5cb5c0929db4e-b76aff0a-79749d99 finished.
New status: syncing-files
[2/7] Processing file mounts
Running `mkdir -p /tmp/ray_tmp_mount/default/~/json && chown -R ubuntu /tmp/ray_tmp_mount/default/~/json`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p /tmp/ray_tmp_mount/default/~/json && chown -R ubuntu /tmp/ray_tmp_mount/default/~/json)'`
Shared connection to 34.69.71.176 closed.
Running `rsync --rsh ssh -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz --exclude **/.git --exclude **/.git/** --filter dir-merge,- .gitignore ./json/ting-1-f3bd25cf2f05.json ubuntu@34.69.71.176:/tmp/ray_tmp_mount/default/~/json/ting-1-f3bd25cf2f05.json`
sending incremental file list
ting-1-f3bd25cf2f05.json
sent 1,764 bytes received 35 bytes 3,598.00 bytes/sec
total size is 2,309 speedup is 1.28
Running `docker inspect -f '{{.State.Running}}' ray_nvidia_docker || true`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker inspect -f '"'"'{{.State.Running}}'"'"' ray_nvidia_docker || true)'`
Shared connection to 34.69.71.176 closed.
`rsync`ed ./json/ting-1-f3bd25cf2f05.json (local) to ~/json/ting-1-f3bd25cf2f05.json (remote)
~/json/ting-1-f3bd25cf2f05.json from ./json/ting-1-f3bd25cf2f05.json
Running `mkdir -p /tmp/ray_tmp_mount/default/~ && chown -R ubuntu /tmp/ray_tmp_mount/default/~`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p /tmp/ray_tmp_mount/default/~ && chown -R ubuntu /tmp/ray_tmp_mount/default/~)'`
Shared connection to 34.69.71.176 closed.
Running `rsync --rsh ssh -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz --exclude **/.git --exclude **/.git/** --filter dir-merge,- .gitignore /tmp/ray-bootstrap-vq2zw790 ubuntu@34.69.71.176:/tmp/ray_tmp_mount/default/~/ray_bootstrap_config.yaml`
sending incremental file list
ray-bootstrap-vq2zw790
sent 1,294 bytes received 35 bytes 2,658.00 bytes/sec
total size is 3,355 speedup is 2.52
Running `docker inspect -f '{{.State.Running}}' ray_nvidia_docker || true`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker inspect -f '"'"'{{.State.Running}}'"'"' ray_nvidia_docker || true)'`
Shared connection to 34.69.71.176 closed.
`rsync`ed /tmp/ray-bootstrap-vq2zw790 (local) to ~/ray_bootstrap_config.yaml (remote)
~/ray_bootstrap_config.yaml from /tmp/ray-bootstrap-vq2zw790
Running `mkdir -p /tmp/ray_tmp_mount/default/~ && chown -R ubuntu /tmp/ray_tmp_mount/default/~`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p /tmp/ray_tmp_mount/default/~ && chown -R ubuntu /tmp/ray_tmp_mount/default/~)'`
Shared connection to 34.69.71.176 closed.
Running `rsync --rsh ssh -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz --exclude **/.git --exclude **/.git/** --filter dir-merge,- .gitignore /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem ubuntu@34.69.71.176:/tmp/ray_tmp_mount/default/~/ray_bootstrap_key.pem`
sending incremental file list
ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem
sent 1,461 bytes received 35 bytes 997.33 bytes/sec
total size is 1,679 speedup is 1.12
Running `docker inspect -f '{{.State.Running}}' ray_nvidia_docker || true`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker inspect -f '"'"'{{.State.Running}}'"'"' ray_nvidia_docker || true)'`
Shared connection to 34.69.71.176 closed.
`rsync`ed /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem (local) to ~/ray_bootstrap_key.pem (remote)
~/ray_bootstrap_key.pem from /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem
[3/7] No worker file mounts to sync
2021-09-06 18:30:47,970 INFO node.py:295 -- wait_for_compute_zone_operation: Waiting for operation operation-1630971047999-5cb5c09f47981-6afc922f-ae8b854d to finish...
2021-09-06 18:30:53,465 INFO node.py:307 -- wait_for_compute_zone_operation: Operation operation-1630971047999-5cb5c09f47981-6afc922f-ae8b854d finished.
New status: setting-up
[4/7] No initialization commands to run.
[5/7] Initalizing command runner
Running `command -v docker || echo 'NoExist'`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (command -v docker || echo '"'"'NoExist'"'"')'`
Shared connection to 34.69.71.176 closed.
Running `docker pull rayproject/ray-ml:latest-gpu`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker pull rayproject/ray-ml:latest-gpu)'`
latest-gpu: Pulling from rayproject/ray-ml
feac53061382: Pull complete
4e74d880cc84: Pull complete
b88d0e36cef0: Pull complete
a4a81bdd364a: Pull complete
f34588647641: Pull complete
cb625e653a5b: Pull complete
8361034ddb18: Pull complete
15971bbf4cfa: Pull complete
71811213304a: Pull complete
ff6d090c8a0a: Pull complete
54e1344d56f5: Pull complete
2324cc0fe838: Pull complete
b5ce686a03b1: Pull complete
4dea84a783ac: Pull complete
fdee1378325f: Pull complete
6c53bcd05a46: Pull complete
2d6f22c6ab1a: Pull complete
d124bc6f7552: Pull complete
f718cbf90ebe: Pull complete
9490fa75f515: Pull complete
0780019b2838: Pull complete
59293d71a1fc: Pull complete
c4623a7ed5da: Pull complete
Digest: sha256:6342f87c9fff0e8391ccf6b491c10d18d42e76c6c06210b4cdbb04ceec03e201
Status: Downloaded newer image for rayproject/ray-ml:latest-gpu
docker.io/rayproject/ray-ml:latest-gpu
Shared connection to 34.69.71.176 closed.
Running `docker inspect -f '{{.State.Running}}' ray_nvidia_docker || true`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker inspect -f '"'"'{{.State.Running}}'"'"' ray_nvidia_docker || true)'`
Shared connection to 34.69.71.176 closed.
Running `docker inspect -f '{{json .Config.Env}}' rayproject/ray-ml:latest-gpu`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker inspect -f '"'"'{{json .Config.Env}}'"'"' rayproject/ray-ml:latest-gpu)'`
Shared connection to 34.69.71.176 closed.
Running `cat /proc/meminfo || true`
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (cat /proc/meminfo || true)'`
Shared connection to 34.69.71.176 closed.
Running `docker info -f '{{.Runtimes}}' `
Full command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker info -f '"'"'{{.Runtimes}}'"'"' )'`
Shared connection to 34.69.71.176 closed.
2021-09-06 18:35:15,293 INFO node.py:295 -- wait_for_compute_zone_operation: Waiting for operation operation-1630971315455-5cb5c19e58578-1f599cdd-5a28bd34 to finish...
2021-09-06 18:35:20,741 INFO node.py:307 -- wait_for_compute_zone_operation: Operation operation-1630971315455-5cb5c19e58578-1f599cdd-5a28bd34 finished.
New status: update-failed
!!!
{'message': 'SSH command failed.'}
SSH command failed.
!!!
Failed to setup head node.
Here is the redo with ssh -vvv.
(env) chris_chiasson@penguin:~/te$ ssh -vvv -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFail
ure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.69.71.176 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker info -f '"'"'{{.Runtimes}}'"'"' )'
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 34.69.71.176 is address
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/ray_ssh_8ec3601252/c21f969b5f/23757eecdedcbfcc38f7fad06d48dddaf9997a16" does not exist
debug2: ssh_connect_direct
debug1: Connecting to 34.69.71.176 [34.69.71.176] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 119914 ms remain after connect
debug1: identity file /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem type -1
debug1: identity file /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 34.69.71.176:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/dev/null"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:5e50WClpGyS28iyXDTFEFb8yE6s4Gf7+eRdSEwHtCQ8
debug3: hostkeys_foreach: reading file "/dev/null"
Warning: Permanently added '34.69.71.176' (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_t-3_ubuntu_0.pem
debug3: sign_and_send_pubkey: RSA SHA256:WkudYX4AaTrHi2TTs1iwbISXMBXTMla66aMhN4UDUbo
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to 34.69.71.176 ([34.69.71.176]:22).
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /tmp/ray_ssh_8ec3601252/c21f969b5f/23757eecdedcbfcc38f7fad06d48dddaf9997a16.aZlJtE3dwH0Mlu3h
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new [/tmp/ray_ssh_8ec3601252/c21f969b5f/23757eecdedcbfcc38f7fad06d48dddaf9997a16]
debug3: muxserver_listen: mux listener channel 0 fd 4
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug1: control_persist_detach: backgrounding master process
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
debug2: set_control_persist_exit_time: schedule exit in 10 seconds
debug2: control_persist_detach: background process is 4878
debug2: fd 4 setting O_NONBLOCK
debug1: multiplexing control connection
debug2: fd 5 setting O_NONBLOCK
debug3: fd 5 is O_NONBLOCK
debug1: channel 1: new [mux-control]
debug3: channel_post_mux_listener: new mux channel 1 fd 5
debug3: mux_master_read_cb: channel 1: hello sent
debug2: set_control_persist_exit_time: cancel scheduled exit
debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
debug2: mux_master_process_hello: channel 1 slave version 4
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
debug2: mux_master_process_alive_check: channel 1: alive check
debug3: mux_client_request_alive: done pid = 4880
debug3: mux_client_request_session: session request sent
debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 200
debug2: mux_master_process_new_session: channel 1: request tty 1, X 0, agent 0, subsys 0, term "xterm-256color", cmd "bash --login -c -i true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker info -f '{{.Runtimes}}' )", env 1
debug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8
debug1: channel 2: new [client-session]
debug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1
debug2: channel 2: send open
debug3: send packet: type 90
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:6: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug1: Sending command: bash --login -c -i true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker info -f '{{.Runtimes}}' )
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug3: mux_session_confirm: sending success reply
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
map[nvidia:{nvidia-container-runtime []} runc:{runc []}]
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: mux_exit_message: channel 2: exit message, exitval 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: channel 2: chan_shutdown_read (i0 o0 sock -1 wfd 6 efd 8 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i3 o1 sock -1 wfd 7 efd 8 [write])
debug2: channel 2: output drain -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: gc: notify user
debug3: mux_master_session_cleanup_cb: entering for channel 2
debug2: channel 1: rcvd close
debug2: channel 1: output open -> drain
debug2: channel 1: chan_shutdown_read (i0 o1 sock 5 wfd 5 efd -1 [closed])
debug2: channel 1: input open -> closed
debug2: channel 2: gc: user detached
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
#1 mux-control (t16 nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1)
#2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1)
debug2: channel 1: obuf empty
debug2: channel 1: chan_shutdown_write (i3 o1 sock 5 wfd 5 efd -1 [closed])
debug2: channel 1: output drain -> closed
debug2: channel 1: is dead (local)
debug2: channel 1: gc: notify user
debug3: mux_master_control_cleanup_cb: entering for channel 1
debug2: channel 1: gc: user detached
debug2: channel 1: is dead (local)
debug2: channel 1: garbage collecting
debug1: channel 1: free: mux-control, nchannels 2
debug3: channel 1: status: The following connections are open:
#1 mux-control (t16 nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1)
debug2: set_control_persist_exit_time: schedule exit in 10 seconds
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
Shared connection to 34.69.71.176 closed.
(env) chris_chiasson@penguin:~/te$ debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug1: ControlPersist timeout expired
debug3: send packet: type 1
debug1: channel 0: free: /tmp/ray_ssh_8ec3601252/c21f969b5f/23757eecdedcbfcc38f7fad06d48dddaf9997a16, nchannels 1
debug3: channel 0: status: The following connections are open:
debug3: fd 0 is not O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
Transferred: sent 2892, received 2488 bytes, in 14.4 seconds
Bytes per second: sent 201.1, received 173.0
debug1: Exit status -1
