Dashboard won't work with on-premise cluster

Dashboard
1

Hello! I am having trouble getting access to the dashboard when running a on-premise cluster with ray.

I am able to use the dashboard with single node python scripts that use ray on my local machine, but when I try the link that the ray cluster launcher gives me upon launching my cluster, it never works :pensive:.

I am refreshing as the nodes are running and still chrome just says 127.0.0.1 refused to connect. I was under the impression that ray takes care of the port forwarding, so I shouldn’t have to be on the same network as the cluster machines to view the dashboard, correct?

Any suggestions appreciated!

2

If this helps, I am running ray on LSF and to access dashboard one needs to port forward on local machine. for reference sharing with you documentation, please refer port forwarding section: GitHub - IBMSpectrumComputing/ray-integration: Ray with LSF. Users can start up a Ray cluster on LSF, and run DL workloads through that either in a batch or interactive mode..

Thanks

3

Thanks for the reference! It did work by doing this on my laptop, but not on my pc where I would like to see it from.

My situation is I want to access the dashboard locally, from my browser. In order to get to the on premise cluster, I first have to ssh into my schools public ssh server, from which I can then ssh into one of the nodes and start the cluster.

The steps you gave were:

export PORT=3752
export HEAD_NODE=ccc2-10.sl.cloud.ibm.com
ssh -L $PORT:localhost:$PORT -N -f -l <username> $HEAD_NODE

The command I run on my laptop to get this to work is
ssh -L 8265:localhost:8265 -N -f -l phesse001 ssh.csbsju.edu

For whatever reason, this command does not work on my pc.

Any suggested debugging tips?

4

In ray 2.0, we made the previous experimental dashboard the new default dashboard. Could you try to upgrade to 2.0 and see if you are still running into this issue?