Problem connecting to GCP cluster

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi,

I’m following the instructions for starting a cluster on GCP with the example-full.yaml config file. It seems to work fine and I can see the head node and worker node VMs running in the GCP console, and I can ssh on to these VMs using the gcloud cmd line tools. However, I can’t connect from my local machine using ray. ray status gives “ConnectionError: Could not find any running Ray instance” and specifying an address doesn’t help.

This is last bit of output I get when starting the cluster. The references to 127.0.0.1 look a bit odd to me.

  [7/7] Starting the Ray runtime
Did not find any active Ray processes.
Shared connection to 34.91.10.196 closed.
Usage stats collection is disabled.

Local node IP: 10.164.0.43

--------------------
Ray runtime started.
--------------------

Next steps
  To add another node to this Ray cluster, run
    ray start --address='10.164.0.43:6379'

  To connect to this Ray cluster:
    import ray
    ray.init()

  To submit a Ray job using the Ray Jobs CLI:
    RAY_ADDRESS='http://127.0.0.1:8265' ray job submit --working-dir . -- python my_script.py

  See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
  for more information on submitting Ray jobs to the Ray cluster.

  To terminate the Ray runtime, run
    ray stop

  To view the status of the cluster, use
    ray status

  To monitor and debug Ray, view the dashboard at
    127.0.0.1:8265
2

I think you need to find your head node’s public IP on GCP and reference that + port number.

3

Thanks. I think it must have been some local firewall rules, I needed to set up some port-forwarding.