# Embedding Grafana visualizations into Ray Dashboard **URL:** \ **Category:** Dashboard, Monitoring & Debugging\ **Created:** [September 6, 2023, 2:10am UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044 "2023-09-06T02:10:58Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![davidchung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidchung/32/5012_2.png) [@davidchung](https://discuss.ray.io/u/davidchung)\ **Post date:** [September 6, 2023, 2:10am UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/1 "2023-09-06T02:10:58Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. Hi! I’ve set up a Ray head node in Docker but am running into some issues with embedding Grafana visualizations into the dashboard. I have separate servers for both Prometheus and Grafana. As far as I can tell these are working as expected, and I can see metrics about the head node in Grafana. The dashboard also seems to be working, except the metrics section says “Set up Prometheus and Grafana for better Ray Dashboard experience” Some things I’ve done: - Set `RAY_GRAFANA_HOST` with protocol/no trailing slash - Set `RAY_PROMETHEUS_HOST ` with protocol/no trailing slash - Set `RAY_GRAFANA_IFRAME_HOST` (Might have done this one incorrectly, I set it to my-grafana-host/d/rayDefaultDashboard/dashboard-name) - In Grafana, under [security] I’ve set allow\_embedding/cookie\_secure to true, and cookie\_samesite to none - Enabled anonymous access (is there a way to pass credentials from Ray to Grafana?) When I go to the /api/grafana\_health endpoint, it returns a 200 response and says “Grafana running” with the correct grafanaHost. However, for the dashboardUids section, I don’t see my dashboard, maybe that’s my issue? Is there an environment variable or something I can set to change this? Or maybe I’m missing a step somewhere? Also ran `grep -r 'grafana' /tmp/ray/sesion_latest/logs/`, but that didn’t provide anything useful either. Thank you! ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [September 6, 2023, 4:58am UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/2 "2023-09-06T04:58:16Z")
Maybe take a look at this PR which is trying to improve doc about this: [polish observability (o11y) docs by scottsun94 · Pull Request #39069 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/39069) ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [September 6, 2023, 5:04am UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/3 "2023-09-06T05:04:01Z")
> [@davidchung](#): > > Set `RAY_GRAFANA_IFRAME_HOST` (Might have done this one incorrectly, I set it to my-grafana-host/d/rayDefaultDashboard/dashboard-name) This one doesn’t seem right. It should be the IP where your browser (and the underlying machine) can reach the grafana server. ---
**Author:** ![davidchung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidchung/32/5012_2.png) [@davidchung](https://discuss.ray.io/u/davidchung)\ **Post date:** [September 6, 2023, 4:50pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/4 "2023-09-06T16:50:12Z")
> Blockquote > This one doesn’t seem right. It should be the IP where your browser (and the underlying machine) can reach the grafana server. So that would be IP of `RAY_GRAFANA_HOST` right? It didn’t seem to like that either 🙁 ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [September 6, 2023, 5:31pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/5 "2023-09-06T17:31:58Z")
Yeah. As long as your browser can reach it. Can you try access IP of `RAY_GRAFANA_HOST` directly in your browser? ---
**Author:** ![davidchung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidchung/32/5012_2.png) [@davidchung](https://discuss.ray.io/u/davidchung)\ **Post date:** [September 6, 2023, 5:42pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/6 "2023-09-06T17:42:37Z")
Yup that seems to work. Going onto the Docker container Ray is running on and curling the Grafana server works as well. ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [September 6, 2023, 6:31pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/7 "2023-09-06T18:31:51Z")
> [@davidchung](#): > > /api/grafana\_health So both the grafana and prometheus healthchecks passed for you? ![Screenshot 2023-09-06 at 11.20.14 AM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/8/8145945e7120ec968e66974e98351e49a93338fb.png) ![Screenshot 2023-09-06 at 11.20.03 AM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/3/3db0aa23c7bf0ee04157061ea36d23ecaa3e81bf.png) ---
**Author:** ![davidchung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidchung/32/5012_2.png) [@davidchung](https://discuss.ray.io/u/davidchung)\ **Post date:** [September 6, 2023, 6:52pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/8 "2023-09-06T18:52:43Z")
Oh that might be it! It was failing cause I had a self-signed certificate, but I fixed that, and now it’s failing because I have authentication turned on for Prometheus. Is there a way to pass credentials or some kind of token to Ray? ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [September 6, 2023, 8:27pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/9 "2023-09-06T20:27:15Z")
Hmm I’m not sure. Can you try turning off the auth for Prometheus first and see if it works? Another thing to note, if you use an existing grafana, you may need to import Ray-provided dashboard JSON into it first: find them after you start Ray cluster at `/tmp/ray/session_latest/metrics/grafana/dashboards` and [copy the JSONs over and import the Grafana dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard) ---
**Author:** ![davidchung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidchung/32/5012_2.png) [@davidchung](https://discuss.ray.io/u/davidchung)\ **Post date:** [September 6, 2023, 8:34pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/10 "2023-09-06T20:34:05Z")
Turning off authentication for Prometheus did the trick. It would be nice to have authentication turned on but at least the visualizations are being embedded, thanks! ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [September 7, 2023, 2:37am UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/11 "2023-09-07T02:37:21Z")
Great to know that. Any idea how we can support it? How can Ray Head Node automatically authenticate to Prometheus in this case? With some credential files, env var or? cc: @aguo ---
**Author:** ![psydok](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/psydok/32/3518_2.png) [@psydok](https://discuss.ray.io/u/psydok)\ **Post date:** [February 6, 2024, 6:01am UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/12 "2024-02-06T06:01:24Z")
> - Enabled anonymous access (is there a way to pass credentials from Ray to Grafana?) Can you please tell me if you have found a solution without enabling anonymous mode? ---
**Author:** ![davidchung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidchung/32/5012_2.png) [@davidchung](https://discuss.ray.io/u/davidchung)\ **Post date:** [February 7, 2024, 10:28pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/13 "2024-02-07T22:28:30Z")
Unfortunately I didn’t 😕 ---
**Author:** ![ayushraj](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@ayushraj](https://discuss.ray.io/u/ayushraj)\ **Post date:** [July 22, 2024, 3:19pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/14 "2024-07-22T15:19:18Z")
@Huaiwei_Sun I have an existing grafana which is hosted in a separate AKS. Is there some way to provide authentication details which are required in my existing Grafana instance ? If not, what are my alternatives if I want to use the same Grafana instance ? Currently I get the **“refused to connect”** error in my Ray dashboard. ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [July 22, 2024, 5:44pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/15 "2024-07-22T17:44:37Z")
Unfortunately I don’t have a good answer to this. I don’t think it currently supports this kind of authentication. Will let @Sam_Chan @aguo confirm. ---
**Author:** ![aguo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/aguo/32/2558_2.png) [@aguo](https://discuss.ray.io/u/aguo)\ **Post date:** [July 22, 2024, 6:13pm UTC](https://discuss.ray.io/t/embedding-grafana-visualizations-into-ray-dashboard/12044/16 "2024-07-22T18:13:14Z")
Unfortunately, i don’t think Grafana supports embedding with authentication: [How to embed Grafana dashboards into web applications | Grafana Labs](https://grafana.com/blog/2023/10/10/how-to-embed-grafana-dashboards-into-web-applications/#not-supported-fully-embedded-private-dashboards-with-authentication) They list out a few reasons there but the primary one is that having to separately authenticate can be a poor experience. For querying prometheus via authentication, I think passing in an API key via env vars would make sense. We’re happy to accept PRs adding this functionality, but it may not be that useful without Grafana supporting authentication.