1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.
2. Environment:
- Ray version: 2.47.1
- Python version: 3.12.11
- OS: Rocky8 (container)
- Cloud/Infrastructure: K8s v1.27.16
- Other libs/tools (if relevant):
- Nvidia H100 with MIG enabled
- CUDA 12.4
- Driver Version 550.54.15
3. Repro steps / sample code: (optional, but helps a lot!)
This happens with manually deployed Ray con K8s. The general setup:
- Head node: no logical resource (head-node)
- CPU-only worker nodes (CPU-node)
- GPU worker nodes with full GPU, no slice (GPU-node)
- GPU worker nodes with MIG-sliced GPU (MIG-node)
In the dashboard Cluster page, head-node, CPU-node, and GPU-node all show information as expected (node name, cpu/mem/gpu/network util, IP,…). Raylet, actors, worker processes are all shown properly.
MIG-nodes on the other hand show no such information. There is only node ID, log, logical resources, labels. Any utilization information is shown as N/A or at 0%, including node name, CPU, GPU, memory, network, IP, disk. Clicking on Node ID show a blank page. There is no information about Raylet, actors, worker processes. However, log access still works. These are the errors shown in the dashboard_agent.log (abbreviated)
2025-07-04 05:40:24,871 ERROR reporter_agent.py:1336 -- Error publishing node physical stats.
Traceback (most recent call last):
File "/src/.venv/lib/python3.12/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1325, in _run_loop
json_payload = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.12/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1343, in _compose_stats_payload
stats = self._collect_stats()
^^^^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.12/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 823, in _collect_stats
"gpus": self._get_gpu_usage(),
^^^^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.12/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 558, in _get_gpu_usage
memory_info = pynvml.nvmlDeviceGetMemoryInfo(gpu_handle)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.12/site-packages/ray/_private/thirdparty/pynvml/pynvml.py", line 2063, in nvmlDeviceGetMemoryInfo
_nvmlCheckReturn(ret)
File "/src/.venv/lib/python3.12/site-packages/ray/_private/thirdparty/pynvml/pynvml.py", line 765, in _nvmlCheckReturn
raise NVMLError(ret)
ray._private.thirdparty.pynvml.pynvml.NVMLError_NoPermission: Insufficient Permissions
It turns out that in order to get the util information from a MIG slice, a different API must be used instead of the usual one. As of latest Ray 2.47.1, the normal API pynvml.nvmlDeviceGetMemoryInfo is used which raises unhandled Permission Error since this API query parent device, not MIG slice. This error crashes ReportHead, so no utilization information is sent.
Another API should be used instead to get the MIG slice handle first, not the parent device handle. Then other query API can be used on the MIG slice handle. However, nvmlDeviceGetUtilizationRates to get compute utilization is not supported for MIG slice, but processes, memory utilization will still work. Try this snippet
# Embedded pynvml in Ray
import ray._private.thirdparty.pynvml as pynvml
pynvml.nvmlInit()
# this is how Ray current does it.
gpu_handle = pynvml.nvmlDeviceGetHandleByIndex(0)
pynvml.nvmlDeviceGetMemoryInfo(gpu_handle) # this raises Permission Error if the device is a MIG slice.
# Instead, get the MIG handle by ID from the parent GPU's handle
# and use this in place of the usual util calls
mig_handle = pynvml.nvmlDeviceGetMigDeviceHandleByIndex(gpu_handle, 0)
# this still does not work, but it is documented to be unsupported.
pynvml.nvmlDeviceGetUtilizationRates(mig_handle)
# these will work though
pynvml.nvmlDeviceGetComputeRunningProcesses(mig_handle)
pynvml.nvmlDeviceGetGraphicsRunningProcesses(mig_handle)
pynvml.nvmlDeviceGetName(mig_handle)
pynvml.nvmlDeviceGetUUID(mig_handle)
Beside the issue with dashboard, important things like scheduling seem to still work.
4. What happened vs. what you expected:
- Expected: Full node info is shown in dashboard for nodes with MIG-sliced Nvidia GPU.
- Actual: Limited to no node info is shown in dashboard for nodes with MIG-sliced Nvidia GPU.