Metrics' Dashboards/panels Not Found

Dashboard
1

Using ray 2.32.0. I set up grafana and prometheus. I’m getting,
“Invalid dashboard UID in annotation request” on the metrics panels and accessing grafana directly shows that there are no dashboards, and when selecting “view in grafana” from the Ray metrics page, grafana shows “Dashboard not found.”

Where is the dashboard

2

How did you setup grafana and prometheus?

3

prometheus lanched via, ray metrics launch-prometheus

grafana via docker container:

version: '3.8'
volumes:
  grafana-storage:
    driver: local

services:
  grafana:
    image: grafana/grafana:latest
    container_name: ray_grafana
    ports:
      - "3001:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
      - ./dashboards:/var/lib/grafana/dashboards
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_SECURITY_ALLOW_EMBEDDING=true 
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
    restart: on-failure:5
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - NET_RAW
      - MKNOD
4

when launching your ray cluster what port are you binding it to (or just leaving it to default)? what compute infra are you running this on; are you just on local machine?

5

Here is my start script:

#!/bin/bash
export RAY_ENABLE_RECORD_ACTOR_TASK_LOGGING=1
export RAY_worker_idle_timeout_milliseconds=30000
export RAY_record_ref_creation_sites=1
export RAY_GRAFANA_HOST=http://192.168.x.x:3001
ray stop
sudo rm -r /ray/tmp
ray start --head --node-ip-address=192.168.x.x --dashboard-host=0.0.0.0 --metrics-export-port 8080

This is being tested from the head node, locally, without any workers.