When I launch Ray on Slurm using the script in the docs I am unable to access the ray dashboard from the url printed to output (Browser just hangs).
I know which machine is the head and I can ssh into it. Is there a way for me to access the dashboard?
@rkube Have you had any issues with accessing the Ray dashboard when running Ray on a SLURM-managed cluster? Any special hoops that one needs to jump through?
What fixed it for me: setting dashboard_host: 0.0.0.0 in ray.init(). I’m not sure about the security of this method though, since it allows all incoming traffic to the slurm compute node. However, that was the only way I’ve been able to view ray dashboard of a remote compute node.
Of course, you have to do port forwarding from remote slurm compute to your local computer, but there’s a good reference here about how to do that.