Okay I had a bit of a play around and I think I know my issue now.
I am running Ray Dashboard on a databricks cluster.
I think from looking at the code what Ray is doing is first to ping the RAY_GRAFANA_HOST/api/health endpoint then expose the graph on the UI.
When accessing Grafana from my web-browser, given the way the Databricks driver proxy is setup, I need to hit:
https://dbc-dp-<my_env_number>.cloud.databricks.com/driver-proxy/o/<my_account>/<my_cluster>/<port>/<grafana_page_link>
If I set RAY_GRAFANA_HOST to the Databricks proxy then it’ll error out on the health check as Databricks will bounce the request from ray dashboard to a login service first that Ray Dashboard cannot handle.
But if I don’t set RAY_GRAFANA_HOST then the health check passes and Ray Dashboard loads the widgets but my local browser can’t load localhost:3000 as the proxy requires It to hit: https://dbc-dp-<my_env_number>.cloud.databricks.com/driver-proxy/o/<my_account>/<my_cluster>/<port>/ as the root address.
Any ideas?