# \[Dashboard\] Missing Physical Resources

**URL:** <https://discuss.ray.io/t/dashboard-missing-physical-resources/13404>\
**Category:** Dashboard, Monitoring & Debugging\
**Created:** [January 13, 2024, 3:49am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404 "2024-01-13T03:49:29Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![matt7](https://avatars.discourse-cdn.com/v4/letter/m/9fc29f/32.png) [@matt7](https://discuss.ray.io/u/matt7)\
**Post date:** [January 13, 2024, 3:49am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/1 "2024-01-13T03:49:29Z")

</div>

Hey guys, I’m new to using Ray and found the Dashboard to be very exciting - I started to set it up with Prometheus + Grafana by following this guide: [Configuring and Managing Ray Dashboard — Ray 2.43.0](https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html#observability-visualization-setup)

On my end I can successfully view the Dashboard, so it seems Grafana, Prometheus, and Ray all successfully work in conjunction.

The issue is that while data is reported for all the logical resources, most of the physical resources (Node CPU, Node Memory, Node Disk, etc.) show no data reported for them.

The only physical resource that has data is Node Count.

Would there be any reason the physical resource usage is missing? Viewing their usage was my main motivation to give Ray Dashboard a try.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [January 15, 2024, 8:32am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/2 "2024-01-15T08:32:59Z")

</div>

Ray doesn’t do anything special here. It collects those physical resource usage using some standard libraries and expose them in the corresponding ports to prometheus to scape for.

1. Can you check if you can see the real-time view of the physical resource usage in ray dashboard’s [cluster page](https://docs.ray.io/en/latest/ray-observability/getting-started.html#cluster-view)?
2. Can you check if you can see those metrics in prometheus and grafana?

---

<div class="post-metadata">

**Author:** ![matt7](https://avatars.discourse-cdn.com/v4/letter/m/9fc29f/32.png) [@matt7](https://discuss.ray.io/u/matt7)\
**Post date:** [January 16, 2024, 12:38am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/3 "2024-01-16T00:38:00Z")

</div>

1. In the cluster page, the physical resource usage for CPU shows 0%, and the Memory is simply left blank (shows nothing).

2. The Grafana plots match my dashboard’s plots exactly it seems. The physical resources are all still missing (besides Node Count).

Prometheus is a bit more involved, but I believe the physical metrics are missing too.

- In the temp folder each ray session creates, one can find a _default\_grafana\_dashboard.json_.

- In that file, logical cpu is calculated from _ray\_resources_, which I can successfully query on Prometheus.

- Also in that file, physical memory is calculated from _ray\_node\_mem\_used_ - I cannot query this variable though (Empty query result), and physical cpu is calculated from _ray\_component\_cpu\_percentage_, which queries to 0.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [January 16, 2024, 5:37am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/4 "2024-01-16T05:37:04Z")

</div>

> 1. In the cluster page, the physical resource usage for CPU shows 0%, and the Memory is simply left blank (shows nothing).

It seems that the physical usage was not collected properly? What are the machines, OS, Ray versions are you using?

---

<div class="post-metadata">

**Author:** ![matt7](https://avatars.discourse-cdn.com/v4/letter/m/9fc29f/32.png) [@matt7](https://discuss.ray.io/u/matt7)\
**Post date:** [January 17, 2024, 2:48am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/5 "2024-01-17T02:48:48Z")

</div>

Ray 2.9 on Windows 11.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [January 17, 2024, 3:14am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/6 "2024-01-17T03:14:22Z")

</div>

@sangcho can you help with it?

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [January 18, 2024, 12:30am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/7 "2024-01-18T00:30:31Z")

</div>

Probably related to Windows.

If you look at dashboard.log and dashboard\_agent.log, are there any stacktrace?

---

<div class="post-metadata">

**Author:** ![matt7](https://avatars.discourse-cdn.com/v4/letter/m/9fc29f/32.png) [@matt7](https://discuss.ray.io/u/matt7)\
**Post date:** [January 20, 2024, 6:56am UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/8 "2024-01-20T06:56:58Z")

</div>

In the dashboard\_agent.log there is an error with “publishing node physical stats.”

After a quick search it turns out to be the same exact error described here: [[Dashboard] shows nothing · Issue #41081 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/41081)

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [January 24, 2024, 7:11pm UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/9 "2024-01-24T19:11:35Z")

</div>

That should be fixed though. Can you follow up in that issue and ask? @matt7

---

<div class="post-metadata">

**Author:** ![bananajoe182](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bananajoe182/32/3452_2.png) [@bananajoe182](https://discuss.ray.io/u/bananajoe182)\
**Post date:** [May 7, 2024, 2:30pm UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/10 "2024-05-07T14:30:51Z")

</div>

I have exactly the same problem with Windows 10, . It worked perfectly until version 2.7.1 and with the update to 2.8.1 the problem was introduced.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/9/94a2cc043bd268def708800102d87f7aa75a9e19.png)

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [May 7, 2024, 3:47pm UTC](https://discuss.ray.io/t/dashboard-missing-physical-resources/13404/11 "2024-05-07T15:47:34Z")

</div>

Please report in that issue and tagged the one who closed the issue.
