We have two Ray clusters running on our on-premise nodes via Ray’s coordinator_server
.
I know we can run scripts on each cluster via ray exec
like this:
ray exec cluster1.yaml script1.py
ray exec cluster2.yaml script2.py
But how would we use Ray Client to connect to the desired cluster? Usually we would do something like:
ray.init("ray://<head_node_host>:10001")
But since the coordinator somewhat arbitrarily chooses the head node during cluster start up, how would a Ray Client script know what the head node host is?
It seems like there should be some command like:
ray.init(coordinator="10.0.0.1:20000", cluster="cluster1")
But I haven’t been able to find anything like that in the docs. Is there a way to use Ray Client with coordinator-managed clusters?