I am running Ray 2.3.1 on my Mac Pro. I also have Grafana and Prometheus running on this machine. I have verified that both are working by checking localhost:3000 and localhost:9090, respectively. I launch a local Ray cluster like so
export RAY_GRAFANA_HOST=http://127.0.0.1:3000
export RAY_PROMETHEUS_HOST=http://127.0.0.1:9090
ray start --head
Ray starts. The Ray monitor at 127.0.0.1:8265 shows broken cluster monitoring windows. The screen looks like this
If I hover over one of the windows I see the message â127.0.0.1 refused to connectâ.
The Ray cluster itself works correctly, as does the Recent jobs tab of the monitor.
I have tried adding export RAY_GRAFANA_IFRAME_HOST=http://127.0.0.1:3000, as well as not setting any of these environment variables, and see the same result.
I watched the web traffic with Chrome developer tools while refreshing the Ray monitor web page. The following things looked wrong:
Two calls to roboto-latin.500 on the Ray monitor port failed with the message âFailed to load response data. No data found for resource for given identifierâ in the Response tab.
Two calls to default-dashboard?... on the Grafana port showed the message âFailed to load response data: No content available because this request was redirectedâ in the Response tab.
Two calls to login on the Grafana port showed the message âFailed to load response data. No resource with the given identifier foundâ in the Response tab.
How do I get Grafana and Prometheus to integrate with Ray?
To the best of my knowledge I followed the documentation instructions you linked to correctly.
Iâll try running ray dashboard <cluster config file>, but I donât know where my cluster config file is. Iâm just having Ray create a local cluster by default.
That worked. The Cluster Utilization and Node Count windows now display data.
For reference of anybody else who hits this, here is exactly how I made this work on my Mac.
brew install grafana
brew install prometheus
Change the --config-file line in /usr/local/etc/prometheus.args to read --config.file /tmp/ray/session_latest/metrics/prometheus/prometheus.yml.
Uncomment the appropriate lines in /usr/local/etc/grafana/grafana.ini so that it matches the contents of /tmp/ray/session_latest/metrics/grafana/grafana.ini.
./prometheus --config.file=/tmp/ray/session_latest/metrics/prometheus/prometheus.yml
grafana-server --config /tmp/ray/session_latest/metrics/grafana/grafana.ini web
My first question is on node machine, embedding metic web cannot display charts, but head can. How to set make node web display right?
My second question is when use docker-compose pull up container of grafana and promethus. Ray dashboard embedding metic web part cannot found any chart. List docker-compose.yml file what used. Seems like network didnât set right.
My first question is on node machine, embedding metic web cannot display charts, but head can. How to set make node web display right?
Iâm not sure if I understand your questions. Can you elaborate? All the env variables need to be set up on head node and the dashboard process is run on head node.
My second question is when use docker-compose pull up container of grafana and promethus.
Check out the setup guide and the requirements here Configuring and Managing Ray Dashboard â Ray 2.5.1. We cannot cover different ways to install/run grafana/prometheus but as long as the setup meets the requirements listed in the documentation, it should work. Let us know if you still run into issues.
I have met that problem on ray 2.4.0. In LAN network, except head node, every machine which could access head node could use web brower access dashboard, but grafana embedding part cannot display. I will down grade later to reproduce the problem.
I donâ't know what ray 2.5.0 changed on dashboard. A new error. Please look at my screenshot.
Please use the head node to access Ray Dashboard. I donât think it works on worker node⌠cc: @sangcho
For the issue itself, we embed the Grafana page to the dashboard. So your child node probably is not able to access the embedded Grafana. As @Huaiwei_Sun said, this is not very well supported use case (but if youâd like to fix it, you should make sure the child node can access the Grafana UI).