Running ray status doesn't show usage when my cluster started with global ip

1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I started a Ray cluster with the following command The startup itself seemed to work, but when I ran ray status, I noticed that nothing was displayed in the Usage section.

RAY_TLS_SERVER_CERT=/home/krouton/certificate.crt RAY_TLS_SERVER_KEY=/home/krouton/private.key RAY_TLS_CA_CERT=/home/krouton/ca_bundle.crt ray start --head --node-ip-address="<the-global-ip-for-this-machine>" --port="20000"  --include-dashboard=False --disable-usage-stats

Instead of the global IP address specified in -node-ip-address, I specified the local IP address of the NIC this machine has and it worked. However, I want to start the Ray head node with a global IP address. How should I do this? Is it correct to start Ray’s head node with a global IP address in the first place?

What I have confirmed

  • Connect the worker node with ray start -address="<the-global-ip-for-this-machine>:20000", and then run Ray runtime started. is displayed on the screen.
  • In that state, I confirmed that headnoding ray status does not show anything in the Usage section.
  • Furthermore, when I run ray status on a worker node, the UUIDs of the nodes in the Healthy section are all gone.
  • If I then run ray status on the head node, No cluster status. is displayed.
  • I confirmed that Ray runtime started. is displayed when the worker node is connected again in that state.
2

Oops, I forgot to submit the results of my commands run.

$ RAY_TLS_SERVER_CERT=/home/krouton/certificate.crt RAY_TLS_SERVER_KEY=/home/krouton/private.key RAY_TLS_CA_CERT=/home/krouton/ca_bundle.crt ray start --head --node-ip-address="<the-global-ip-for-this-machine>" --port="20000"  --include-dashboard=False --disable-usage-stats
Usage stats collection is disabled.

Local node IP: <the-global-ip-for-this-machine>

--------------------
Ray runtime started.
--------------------

Next steps
  To connect to this Ray runtime from another node, run
    ray start --address='<the-global-ip-for-this-machine>:20000'

  Alternatively, use the following Python code:
    import ray
    ray.init(address='auto', _node_ip_address='<the-global-ip-for-this-machine>')

  To connect to this Ray runtime from outside of the cluster, for example to
  connect to a remote cluster from your laptop directly, use the following
  Python code:
    import ray
    ray.init(address='ray://<head_node_ip_address>:None')

  If connection fails, check your firewall settings and network configuration.

  To terminate the Ray runtime, run
    ray stop

$ ray status
======== Autoscaler status: 2023-01-13 03:24:40.005890 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_79a1abc083c7a86fe6e9266ff000a40ed2896a51b4e8449ba167ada4
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:


Demands:
 (no resource demands)
3

Further to what I have confirmed.

  • If I set -node-ip-address to the local IP address, it works.
  • If I set RAY_USE_TLS=0 and use the global IP address, it works.