# Ray Monitor Not Connecting to Grafana and Prometheus

**URL:** <https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233>\
**Category:** Dashboard, Monitoring & Debugging\
**Created:** [April 14, 2023, 7:30pm UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233 "2023-04-14T19:30:56Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wpm/32/4241_2.png) [@wpm](https://discuss.ray.io/u/wpm)\
**Post date:** [April 14, 2023, 7:30pm UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/1 "2023-04-14T19:30:56Z")

</div>

I am running Ray 2.3.1 on my Mac Pro. I also have Grafana and Prometheus running on this machine. I have verified that both are working by checking localhost:3000 and localhost:9090, respectively. I launch a local Ray cluster like so

```auto
export RAY_GRAFANA_HOST=http://127.0.0.1:3000
export RAY_PROMETHEUS_HOST=http://127.0.0.1:9090
ray start --head

```

Ray starts. The Ray monitor at 127.0.0.1:8265 shows broken cluster monitoring windows. The screen looks like this

 ![ray-monitor](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/d/db1c9aa67cd30c140ce6e123588afad60a941ad6.png)

If I hover over one of the windows I see the message “127.0.0.1 refused to connect”.

The Ray cluster itself works correctly, as does the Recent jobs tab of the monitor.

I have tried adding `export RAY_GRAFANA_IFRAME_HOST=http://127.0.0.1:3000`, as well as not setting any of these environment variables, and see the same result.

I watched the web traffic with Chrome developer tools while refreshing the Ray monitor web page. The following things looked wrong:

- Two calls to `roboto-latin.500` on the Ray monitor port failed with the message “Failed to load response data. No data found for resource for given identifier” in the Response tab.
- Two calls to `default-dashboard?...` on the Grafana port showed the message “Failed to load response data: No content available because this request was redirected” in the Response tab.
- Two calls to `login` on the Grafana port showed the message “Failed to load response data. No resource with the given identifier found” in the Response tab.

How do I get Grafana and Prometheus to integrate with Ray?

---

<div class="post-metadata">

**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\
**Post date:** [April 15, 2023, 1:19am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/2 "2023-04-15T01:19:31Z")

</div>

@rickyyx @sangcho Do we have [specific instructions](https://docs.ray.io/en/latest/cluster/running-applications/monitoring-and-observability.html#cluster-monitoring) how to [install Grafana and Prometheus on local host](https://docs.ray.io/en/latest/ray-observability/ray-metrics.html#deploying-grafana) and how Ray dashboard can discover its configs?

Does @wpm have to use the dashboard command: `ray dashboard [-p <port, 8265 by default>] <cluster config file>`

---

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wpm/32/4241_2.png) [@wpm](https://discuss.ray.io/u/wpm)\
**Post date:** [April 15, 2023, 4:28am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/3 "2023-04-15T04:28:49Z")

</div>

To the best of my knowledge I followed the documentation instructions you linked to correctly.

I’ll try running `ray dashboard <cluster config file>`, but I don’t know where my cluster config file is. I’m just having Ray create a local cluster by default.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 15, 2023, 4:52am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/4 "2023-04-15T04:52:47Z")

</div>

`ray dashboard` is not needed for a local ray cluster. Hmm. I just tried to set those up on my macbook pro, it worked fine.

- What’s your grafana and prometheus version?
- What commands did you use to launch them?
- Can you see the metrics in the grafana? There should be a grafana dashboard called “Default Dashboard”  
 ![Screen Shot 2023-04-14 at 9.52.02 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/4/471978c5671922a01515a68f8a3731ea7c286eb7.png)

---

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wpm/32/4241_2.png) [@wpm](https://discuss.ray.io/u/wpm)\
**Post date:** [April 15, 2023, 5:04am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/5 "2023-04-15T05:04:50Z")

</div>

- Grafana 9.4.7
- Prometheus 2.43.0
- Grafana started with `brew services start grafana`
- Prometheus started with `docker run -p 9090:9090 prom/prometheus`
- I don’t think I have any dashboards in Grafana. I poked around the UI and didn’t see anything.

I have a dashboards page that looks like this

 ![grafana](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/7/706c2d5d9f38bdab4f886f0c8c9fe9f62a6332c3.png)

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 15, 2023, 5:19am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/6 "2023-04-15T05:19:35Z")

</div>

According to: [Metrics — Ray 2.3.1](https://docs.ray.io/en/latest/ray-observability/ray-metrics.html#getting-started)

 ![Screen Shot 2023-04-14 at 10.16.23 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/b/b8f2a6be629d69ef20051e6a754e1c56f6238977.png)  
 ![Screen Shot 2023-04-14 at 10.16.30 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/d/dc3dfcb5b473590469fa547c97348a779e413dc3.png)

You need to start prometheus and grafana with the config files provided by Ray so that:

- prometheus can scrap the metrics from the ray cluster properly
- grafana can talk to the prometheus and visualize the metrics with the template dashboard provided by Ray

Can you give it a try?

---

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wpm/32/4241_2.png) [@wpm](https://discuss.ray.io/u/wpm)\
**Post date:** [April 15, 2023, 5:59pm UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/7 "2023-04-15T17:59:19Z")

</div>

That worked. The Cluster Utilization and Node Count windows now display data.

For reference of anybody else who hits this, here is exactly how I made this work on my Mac.

1. `brew install grafana`

2. `brew install prometheus`

3. Change the `--config-file line` in `/usr/local/etc/prometheus.args` to read `--config.file /tmp/ray/session_latest/metrics/prometheus/prometheus.yml`.

4. Uncomment the appropriate lines in `/usr/local/etc/grafana/grafana.ini` so that it matches the contents of `/tmp/ray/session_latest/metrics/grafana/grafana.ini`.

5. `brew services start grafana`

6. `brew services start prometheus`

7. `ray start --head`

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 15, 2023, 6:47pm UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/8 "2023-04-15T18:47:11Z")

</div>

Glad that it works out!

@aguo We probably should add some guides for homebrew-based workflows ^. Added it to our backlog.

---

<div class="post-metadata">

**Author:** ![funk\_Jz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/funk_jz/32/4568_2.png) [@funk\_Jz](https://discuss.ray.io/u/funk_Jz)\
**Post date:** [June 16, 2023, 8:22am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/9 "2023-06-16T08:22:58Z")

</div>

Command line mode could work like this.

```auto
./prometheus --config.file=/tmp/ray/session_latest/metrics/prometheus/prometheus.yml

grafana-server --config /tmp/ray/session_latest/metrics/grafana/grafana.ini web

```

My first question is on node machine, embedding metic web cannot display charts, but head can. How to set make node web display right?

My second question is when use docker-compose pull up container of grafana and promethus. Ray dashboard embedding metic web part cannot found any chart. List docker-compose.yml file what used. Seems like network didn’t set right.

```auto
version: '3'

networks:
    ray_dashboard:
        driver: bridge

services:
    prometheus:
        image: prom/prometheus
        container_name: prometheus
        hostname: prometheus
        restart: always
        volumes:
            - /tmp/ray/session_latest/metrics/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
        ports:
            - "9090:9090"
        networks:
            - ray_dashboard
        # network_mode: "host"

    grafana:
        image: grafana/grafana
        container_name: grafana
        hostname: grafana
        restart: always
        # environment:
          # GF_PATHS_CONFIG: /tmp/ray/session_latest/metrics/grafana/grafana.ini
        volumes:
            - /tmp/ray/session_latest/metrics/grafana/grafana.ini:/etc/grafana/grafana.ini
        ports:
            - "3000:3000"
        networks:
            - ray_dashboard

```

Official user guide didn’t give information about this. Docker compose is more convient than download grafana and promethus respectly.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [June 26, 2023, 4:20am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/10 "2023-06-26T04:20:44Z")

</div>

> My first question is on node machine, embedding metic web cannot display charts, but head can. How to set make node web display right?

I’m not sure if I understand your questions. Can you elaborate? All the env variables need to be set up on head node and the dashboard process is run on head node.

> My second question is when use docker-compose pull up container of grafana and promethus.

Check out the setup guide and the requirements here [Configuring and Managing Ray Dashboard — Ray 2.8.0](https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html). We cannot cover different ways to install/run grafana/prometheus but as long as the setup meets the requirements listed in the documentation, it should work. Let us know if you still run into issues.

---

<div class="post-metadata">

**Author:** ![funk\_Jz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/funk_jz/32/4568_2.png) [@funk\_Jz](https://discuss.ray.io/u/funk_Jz)\
**Post date:** [June 26, 2023, 4:46am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/11 "2023-06-26T04:46:13Z")

</div>

On head node, child node could access dashboad:

```auto
ray start --head --dashboard-host="0.0.0.0"

```

Grafana and premethus run on head node:

```auto
grafana-server --config /tmp/ray/session_latest/metrics/grafana/grafana.ini web

./prometheus --config.file=/tmp/ray/session_latest/metrics/prometheus/prometheus.yml

```

On head node, all work right. On child node, grafana couldn’t display.

For latest grafana OSS version 10.0.0:

```auto
sudo grafana server --config /tmp/ray/session_latest/metrics/grafana/grafana.ini web

```

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [June 26, 2023, 5:17am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/12 "2023-06-26T05:17:49Z")

</div>

> On child node, grafana couldn’t display.

You mean worker node? Whatl do you mean by `grafana couldn’t display`?

---

<div class="post-metadata">

**Author:** ![funk\_Jz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/funk_jz/32/4568_2.png) [@funk\_Jz](https://discuss.ray.io/u/funk_Jz)\
**Post date:** [June 26, 2023, 5:58am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/13 "2023-06-26T05:58:23Z")

</div>

I have met that problem on ray 2.4.0. In LAN network, except head node, every machine which could access head node could use web brower access dashboard, but grafana embedding part cannot display. I will down grade later to reproduce the problem.

I don‘'t know what ray 2.5.0 changed on dashboard. A new error. Please look at my screenshot.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/1/18b1867f493afcd5ec2695bc4c55979084fc8140.png)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/7/7001ed6c0ba34ee870c876d8123d8dce2043e2c3.png)

---

<div class="post-metadata">

**Author:** ![funk\_Jz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/funk_jz/32/4568_2.png) [@funk\_Jz](https://discuss.ray.io/u/funk_Jz)\
**Post date:** [June 26, 2023, 6:25am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/14 "2023-06-26T06:25:56Z")

</div>

Let’s go back dashboard issue.

On child node(192.168.0.104):

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/f/f59e6fcfbb8b31c2507156ade87f4fca17930d36.png)

---

<div class="post-metadata">

**Author:** ![funk\_Jz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/funk_jz/32/4568_2.png) [@funk\_Jz](https://discuss.ray.io/u/funk_Jz)\
**Post date:** [June 26, 2023, 6:27am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/15 "2023-06-26T06:27:15Z")

</div>

On head node(192.168.0.102):

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/7/7948946024cba5e71938685a0d7a9735f8271946.png)

Here mention change environment variable RAY\_GRAFANA\_HOST and RAY\_PROMETHEUS\_HOST.  
[https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html#embed-grafana-in-dashboard](https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html#embed-grafana-in-dashboard)

But in this document, cli mode start cluster didn’t gave us an example about how to set environment variable in runtime:  
[https://docs.ray.io/en/latest/ray-core/starting-ray.html#start-ray-cli](https://docs.ray.io/en/latest/ray-core/starting-ray.html#start-ray-cli)

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [June 26, 2023, 7:08am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/16 "2023-06-26T07:08:59Z")

</div>

@funk_Jz the error in 2.5 indicates that `prometheus.yml` is directory. Do you know why? It’s not related to ray but more to your system.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [June 26, 2023, 7:17am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/17 "2023-06-26T07:17:21Z")

</div>

1. Please use the head node to access Ray Dashboard. I don’t think it works on worker node… cc: @sangcho
2. If you don’t use VMs ([Ray on Cloud VMs — Ray 2.5.1](https://docs.ray.io/en/latest/cluster/vms/index.html)) or K8s ([Ray on Kubernetes — Ray 2.5.1](https://docs.ray.io/en/latest/cluster/kubernetes/index.html)) to start ray cluster, try setting the env variable on the head node before starting the cluster manually.

---

<div class="post-metadata">

**Author:** ![funk\_Jz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/funk_jz/32/4568_2.png) [@funk\_Jz](https://discuss.ray.io/u/funk_Jz)\
**Post date:** [June 26, 2023, 7:26am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/18 "2023-06-26T07:26:26Z")

</div>

Haha, I think it is a metaphysics issue(prometheus.yml change to a directory). So clear up the /tmp/ray, and restart cluster. Everything goes well.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/7/7d3e6406437c044a0f6f3b618b3d08aeceb8ef2b.png)

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [June 28, 2023, 12:16am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/19 "2023-06-28T00:16:20Z")

</div>

> 1. Please use the head node to access Ray Dashboard. I don’t think it works on worker node… cc: @sangcho

For the issue itself, we embed the Grafana page to the dashboard. So your child node probably is not able to access the embedded Grafana. As @Huaiwei_Sun said, this is not very well supported use case (but if you’d like to fix it, you should make sure the child node can access the Grafana UI).

---

<div class="post-metadata">

**Author:** ![wpm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wpm/32/4241_2.png) [@wpm](https://discuss.ray.io/u/wpm)\
**Post date:** [January 14, 2024, 2:15am UTC](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233/20 "2024-01-14T02:15:35Z")

</div>

I thought this was solved, but it’s not.

I did the same steps I outlined in my post from April 1, 2023, on a different machine and again seeing the “localhost refused to connect” problem in the Cluster Utilization and Node Count windows.

I am running Prometheus and Grafana from a Homebrew install. They are both working.

```auto
% brew services
Name Status User File
grafana started mcneill ~/Library/LaunchAgents/homebrew.mxcl.grafana.plist
prometheus started mcneill ~/Library/LaunchAgents/homebrew.mxcl.prometheus.plist

```

I can see web pages at [http://localhost:9090](http://localhost:9090) and [http://localhost:3000](http://localhost:3000).

The Prometheus log at `/opt/homebrew/var/log/prometheus.err.log` shows that `/tmp/ray/session_latest/metrics/prometheus/prometheus.yml` has been loaded.

```auto
ts=2024-01-14T01:47:20.741Z caller=main.go:539 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2024-01-14T01:47:20.741Z caller=main.go:583 level=info msg="Starting Prometheus Server" mode=server version="(version=2.48.1, branch=non-git, revision=non-git)"
ts=2024-01-14T01:47:20.741Z caller=main.go:588 level=info build_context="(go=go1.21.5, platform=darwin/arm64, user=brew@Sonoma-arm64.local, date=20231208-09:22:46, tags=netgo,builtinassets,stringlabels)"
ts=2024-01-14T01:47:20.741Z caller=main.go:589 level=info host_details=(darwin)
ts=2024-01-14T01:47:20.741Z caller=main.go:590 level=info fd_limits="(soft=61440, hard=unlimited)"
ts=2024-01-14T01:47:20.741Z caller=main.go:591 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2024-01-14T01:47:20.743Z caller=web.go:566 level=info component=web msg="Start listening for connections" address=127.0.0.1:9090
ts=2024-01-14T01:47:20.743Z caller=main.go:1024 level=info msg="Starting TSDB ..."
ts=2024-01-14T01:47:20.743Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1705177341653 maxt=1705183200000 ulid=01HM2TA7KYTCCA3NT2A39Z3GM4
ts=2024-01-14T01:47:20.743Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1705185811653 maxt=1705190400000 ulid=01HM2TA7XPS25VJ6W7K6MZ7QCN
ts=2024-01-14T01:47:20.743Z caller=tls_config.go:274 level=info component=web msg="Listening on" address=127.0.0.1:9090
ts=2024-01-14T01:47:20.743Z caller=tls_config.go:277 level=info component=web msg="TLS is disabled." http2=false address=127.0.0.1:9090
ts=2024-01-14T01:47:20.745Z caller=head.go:601 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2024-01-14T01:47:20.746Z caller=head.go:682 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=385.208µs
ts=2024-01-14T01:47:20.746Z caller=head.go:690 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2024-01-14T01:47:20.753Z caller=head.go:726 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2024-01-14T01:47:20.754Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=68 maxSegment=72
ts=2024-01-14T01:47:20.755Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=69 maxSegment=72
ts=2024-01-14T01:47:20.767Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=70 maxSegment=72
ts=2024-01-14T01:47:20.767Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=71 maxSegment=72
ts=2024-01-14T01:47:20.767Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=72 maxSegment=72
ts=2024-01-14T01:47:20.767Z caller=head.go:798 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=7.654417ms wal_replay_duration=13.646042ms wbl_replay_duration=42ns total_replay_duration=21.713917ms
ts=2024-01-14T01:47:20.769Z caller=main.go:1045 level=info fs_type=1a
ts=2024-01-14T01:47:20.769Z caller=main.go:1048 level=info msg="TSDB started"
ts=2024-01-14T01:47:20.769Z caller=main.go:1230 level=info msg="Loading configuration file" filename=/tmp/ray/session_latest/metrics/prometheus/prometheus.yml
ts=2024-01-14T01:47:20.787Z caller=main.go:1267 level=info msg="Completed loading of configuration file" filename=/tmp/ray/session_latest/metrics/prometheus/prometheus.yml totalDuration=17.422708ms db_storage=625ns remote_storage=791ns web_handler=208ns query_engine=458ns scrape=17.257833ms scrape_sd=27.292µs notify=791ns notify_sd=1.125µs rules=1.459µs tracing=7.208µs
ts=2024-01-14T01:47:20.787Z caller=main.go:1009 level=info msg="Server is ready to receive web requests."
ts=2024-01-14T01:47:20.787Z caller=manager.go:1012 level=info component="rule manager" msg="Starting rule manager..."

```

My `/usr/local/etc/prometheus.args` file looks like this:

```auto
--config.file /tmp/ray/session_latest/metrics/prometheus/prometheus.yml

```

My `/tmp/ray/session_latest/metrics/prometheus/prometheus.yml` file looks like this:

```yml
# my global config
global:
  scrape_interval: 10s # Set the scrape interval to every 10 seconds. Default is every 1 minute.
  evaluation_interval: 10s # Evaluate rules every 10 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

scrape_configs:
# Scrape from each ray node as defined in the service_discovery.json provided by ray.
- job_name: 'ray'
  file_sd_configs:
  - files:
    - '/tmp/ray/prom_metrics_service_discovery.json'

```

My `/usr/local/etc/grafana/grafana.ini` file looks like this:

```auto
[security]
allow_embedding = true

[auth.anonymous]
enabled = true
org_name = Main Org.
org_role = Viewer

[paths]
provisioning = /tmp/ray/session_latest/metrics/grafana/provisioning

```

What am I doing wrong? Is there anything I can do to debug this. Any error messages anywhere?

[Next page](https://discuss.ray.io/t/ray-monitor-not-connecting-to-grafana-and-prometheus/10233.md?page=2)