How can I override the DEFAULT_GRAFANA_HOST in metric_heads.py?
For my setup, localhost isn’t working and I need to change it to the full link.
I have tried manually inserting the full link in the browser and that works.
Thanks
It is not merged yet, but this can probably helps? Improve documentation for setting up grafana and prometheus by alanwguo · Pull Request #31129 · ray-project/ray · GitHub
Thanks @sangcho I just had a look at the docu and tried setting RAY_GRAFANA_HOST as discussed. It seems my dashboard is still trying to hit localhost?
Does that require a new Ray release too? Or it should work and maybe Ray just isn’t picking up the env var?
Okay I had a bit of a play around and I think I know my issue now.
I am running Ray Dashboard on a databricks cluster.
I think from looking at the code what Ray is doing is first to ping the RAY_GRAFANA_HOST/api/health endpoint then expose the graph on the UI.
When accessing Grafana from my web-browser, given the way the Databricks driver proxy is setup, I need to hit:
https://dbc-dp-<my_env_number>.cloud.databricks.com/driver-proxy/o/<my_account>/<my_cluster>/<port>/<grafana_page_link>
If I set RAY_GRAFANA_HOST to the Databricks proxy then it’ll error out on the health check as Databricks will bounce the request from ray dashboard to a login service first that Ray Dashboard cannot handle.
But if I don’t set RAY_GRAFANA_HOST then the health check passes and Ray Dashboard loads the widgets but my local browser can’t load localhost:3000 as the proxy requires It to hit: https://dbc-dp-<my_env_number>.cloud.databricks.com/driver-proxy/o/<my_account>/<my_cluster>/<port>/ as the root address.
Any ideas?
cc: @aguo for ideas on this
Hi Brian,
We actually have two env vars that so that you can customize both the host the dashboard api server uses to hit grafana and the the url that the browser uses to embed the grafana graphs.
The first, RAY_GRAFANA_HOST can use your localhost value since it sounds like the dashboard api server is running on the same host as your grafana instance.
The second, RAY_GRAFANA_IFRAME_HOST allows you to customize the host used by the browser. This should be set to your https://dbc-db... hostname value.
Let me know if that works!
RAY_GRAFANA_IFRAME_HOST is more a less a “private” configuration since we weren’t sure if users would find it useful. If it is a common usecase, we can make this public and document it.
Thanks @aguo that RAY_GRAFANA_IFRAME_HOST was what I needed and fixes it
Ray==2.6.3
Python==3.10.12
Hi,
Thank you for your dedication in the Ray library !
I am fetching over lan, the ip of the head node where the dashboard is hosted. I can access the ray dashboard with configured prometheus and grafana servers also, but it seems that the ray’s iframe embedding grafana’s metrics has an error in the url as the protocol http:// is missing:
<iframe src="192.168.1.18:3000/d-solo/rayDefaultDashboard/default-dashboard?orgId=1&[...]"
When i modify the code in the front-end it works great with :
<iframe src="http://192.168.1.18:3000/d-solo/rayDefaultDashboard/default-dashboard?orgId=1&[...]"
Configuration:
RAY_GRAFANA_IFRAME_HOST to the IP of the head node 192.168.1.18RAY_GRAFANA/PROMETHEUS_HOST as it seems not required.--dashboard-host 192.168.1.18grafana.ini with :[server]
protocol = http
http_port = 3000
domain = 192.168.1.18
enforce_domain=false
# Tried both
root_url=%(protocol)s://%(domain)s:%(http_port)s/
root_url=http://192.168.1.18:3000
....
I am certainly missing something but at this point i can’t find it.
Should i had some extra configuration to the grafana.ini ?
Thanks
(Not sure if i should reply here or open a new post)
What error did you see?
Have you configured all the settings as per the instruction here? Configuring and Managing Ray Dashboard — Ray 2.6.1