We are troubleshooting why Grafana embedded in Ray monitoring incorrectly queries data via /api/annotations, causing no data to display, and preparing to report this issue to the Ray community.

Dashboard, Monitoring & Debugging
1

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround (e.g., using standalone Grafana).
2. Environment:

  • Ray version: 2.46.0
  • Python version: 3.10.12
  • OS: Ubuntu 22.04
  • Other libs/tools (if relevant): Grafana 12.0.0

3. Repro steps / sample code:

  1. Integrate Ray with an existing or newly installed Grafana instance.
  2. Ensure Ray is configured to embed a specific Grafana dashboard, which in my case appears to be identified in Grafana by dashboardUID=rayDefaultDashboard.
  3. Configure this rayDefaultDashboard in Grafana as follows:
    a. Create (or ensure it has) dashboard variables. One crucial variable is named datasource and is defined with a constant value of “prometheus” (this “prometheus” refers to a configured Grafana data source pointing to a Prometheus instance).
    b. Panels within this dashboard are configured to use ${datasource} as their data source.
    c. The queries within these panels are standard PromQL queries (e.g., sum(increase(ray_tasks{...}[...])) by (State)).
  4. Access the Ray Dashboard UI section where these Grafana panels are embedded.
  5. Open browser developer tools and inspect the network requests made by the embedded Grafana panels.

4. What happened vs. what you expected:

  • Expected:

    • The embedded Grafana panels in the Ray Dashboard should use the configured data source (resolved from the ${datasource} variable to “prometheus”).
    • Network requests for panel data should be made to Grafana’s /api/ds/query endpoint, with PromQL queries targeting the Prometheus data source.
    • Panels should display metric data correctly.

  • Actual:

    • All embedded Grafana panels (from the rayDefaultDashboard) in the Ray Dashboard consistently make network requests to Grafana’s /api/annotations?from=...&to=...&limit=100&matchAny=false&dashboardUID=rayDefaultDashboard endpoint.
    • As a result, the panels show “no data” or do not display the expected metrics.
    • Accessing the same rayDefaultDashboard directly in Grafana (not embedded in Ray) works as expected: panels use the “prometheus” data source, PromQL queries are issued via /api/ds/query, and data is displayed correctly.
    • The grafana.ini settings for allow_embedding = true, auth.anonymous (if used by Ray), and cookie_secure = false (for HTTP access) have been checked and appear correct for allowing embedding and login.

The core issue seems to be that the Ray Dashboard’s mechanism for embedding or fetching data from the specified Grafana dashboard (rayDefaultDashboard) is incorrectly targeting the /api/annotations endpoint for all panels, despite the Grafana dashboard itself (variables, panel queries, data source selection) being configured to use a Prometheus data source with PromQL. This occurs even when the datasource variable is explicitly defined as “prometheus” in the Grafana dashboard settings.

3

dup of RAY Dashboad Grafana Embedding ERROR