How can I override the DEFAULT_GRAFANA_HOST in metric_heads.py?
For my setup, localhost isn’t working and I need to change it to the full link.
I have tried manually inserting the full link in the browser and that works.
Thanks
It is not merged yet, but this can probably helps? Improve documentation for setting up grafana and prometheus by alanwguo · Pull Request #31129 · ray-project/ray · GitHub
Thanks @sangcho I just had a look at the docu and tried setting RAY_GRAFANA_HOST as discussed. It seems my dashboard is still trying to hit localhost?
Does that require a new Ray release too? Or it should work and maybe Ray just isn’t picking up the env var?
Okay I had a bit of a play around and I think I know my issue now.
I am running Ray Dashboard on a databricks cluster.
I think from looking at the code what Ray is doing is first to ping the RAY_GRAFANA_HOST/api/health endpoint then expose the graph on the UI.
When accessing Grafana from my web-browser, given the way the Databricks driver proxy is setup, I need to hit:
https://dbc-dp-<my_env_number>.cloud.databricks.com/driver-proxy/o/<my_account>/<my_cluster>/<port>/<grafana_page_link>
If I set RAY_GRAFANA_HOST to the Databricks proxy then it’ll error out on the health check as Databricks will bounce the request from ray dashboard to a login service first that Ray Dashboard cannot handle.
But if I don’t set RAY_GRAFANA_HOST then the health check passes and Ray Dashboard loads the widgets but my local browser can’t load localhost:3000 as the proxy requires It to hit: https://dbc-dp-<my_env_number>.cloud.databricks.com/driver-proxy/o/<my_account>/<my_cluster>/<port>/ as the root address.
Any ideas?
Hi Brian,
We actually have two env vars that so that you can customize both the host the dashboard api server uses to hit grafana and the the url that the browser uses to embed the grafana graphs.
The first, RAY_GRAFANA_HOST can use your localhost value since it sounds like the dashboard api server is running on the same host as your grafana instance.
The second, RAY_GRAFANA_IFRAME_HOST allows you to customize the host used by the browser. This should be set to your https://dbc-db... hostname value.
Let me know if that works!
RAY_GRAFANA_IFRAME_HOST is more a less a “private” configuration since we weren’t sure if users would find it useful. If it is a common usecase, we can make this public and document it.