Question about reverse ssh connection from worker

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Dear experts,
I hope to ask the question that whether it is possible to use reverse ssh connection when communicate with worker node. In our cluster setup it is not possible to do ssh from head to worker, but the reverse direction is (also allow the port mapping). I had tried to map the GCS port to local in the worker node, though the worker could now connect to head, the head GCS does not recognize and simply marks the worker as dead.

In short my question is whether ray cluster support the case, when only ssh established from worker node is allowed and is it possible to make it work by mapping specific ports?

I am using Ray 2.24.0 with custom autoscaler (similar to the fake_multi_node in official repo)

Thank you for any suggestion or hint.

Best,
Q.

2

I don’t believe Ray uses ssh (as in secure shell protocol), but I guess you don’t mean that and talking about ssh tunneling as a way to get around your connectivity challenges. Ray documentation talks about the need for all cluster notes to have bidirectional communication, so you’d need to do a lot of port forwarding, but it should work.

If you are running into issues and want further advice, you should share the configuration for your head and worker nodes and details about errors you see in logs.

3

Hi thank you for the reply. Yes it is indeed a problem of port forwarding and after fwd all the ports in the right direction (dynamic tunnel helps) the problem was solved.

Best,
Q.