Error computing CPU usage of Ray Kubernetes pod - Docker Desktop

Hi Team, need your help with this.

I am deploying ray on Kubernetes running out of Docker Desktop. I have followed example mentioned at Deploying Ray Serve — Ray v2.0.0.dev0

Here is error logged while ray submit.

❯ ray submit example-full.yaml
2022-01-06 18:07:22,800	INFO -- setting max workers for head node type to 0
Loaded cached provider configuration
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
2022-01-06 18:07:25,511	INFO -- setting max workers for head node type to 0
2022-01-06 18:07:25,531	INFO -- NodeUpdater: example-cluster-ray-head-type-ccmcx: Running kubectl -n ray exec -it example-cluster-ray-head-type-ccmcx -- bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (python ~/'
2022-01-06 18:07:27,416	INFO -- Connecting to existing Ray cluster at address:
(ServeController pid=676) 2022-01-06 18:07:28,921	INFO -- Using RayInternalKVStore for controller checkpoint and recovery.
(ServeController pid=676) 2022-01-06 18:07:28,930	INFO -- Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-node:' on node 'node:' listening on ''
(raylet, ip= --- Logging error ---
(raylet, ip= Traceback (most recent call last):
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/", line 38, in cpu_percent
(raylet, ip=     cpu_usage = _cpu_usage()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/", line 64, in _cpu_usage
(raylet, ip=     return int(open(CPU_USAGE_PATH).read())
(raylet, ip= FileNotFoundError: [Errno 2] No such file or directory: '/sys/fs/cgroup/cpuacct/cpuacct.usage'
(raylet, ip=
(raylet, ip= During handling of the above exception, another exception occurred:
(raylet, ip=
(raylet, ip= Traceback (most recent call last):
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/logging/", line 69, in emit
(raylet, ip=     if self.shouldRollover(record):
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/logging/", line 185, in shouldRollover
(raylet, ip=     msg = "%s\n" % self.format(record)
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/logging/", line 869, in format
(raylet, ip=     return fmt.format(record)
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/logging/", line 608, in format
(raylet, ip=     record.message = record.getMessage()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/logging/", line 369, in getMessage
(raylet, ip=     msg = msg % self.args
(raylet, ip= TypeError: not all arguments converted during string formatting
(raylet, ip= Call stack:
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/", line 376, in <module>
(raylet, ip=     loop.run_until_complete(
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/asyncio/", line 574, in run_until_complete
(raylet, ip=     self.run_forever()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/asyncio/", line 541, in run_forever
(raylet, ip=     self._run_once()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/asyncio/", line 1786, in _run_once
(raylet, ip=     handle._run()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/asyncio/", line 88, in _run
(raylet, ip=, *self._args)
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/modules/reporter/", line 554, in run
(raylet, ip=     await self._perform_iteration(aioredis_client)
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/modules/reporter/", line 539, in _perform_iteration
(raylet, ip=     stats = self._get_all_stats()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/modules/reporter/", line 334, in _get_all_stats
(raylet, ip=     "cpu": self._get_cpu_percent(),
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/modules/reporter/", line 193, in _get_cpu_percent
(raylet, ip=     return k8s_utils.cpu_percent()
(raylet, ip=   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/", line 57, in cpu_percent
(raylet, ip=     logger.exception("Error computing CPU usage of Ray Kubernetes pod.", e)
(raylet, ip= Message: 'Error computing CPU usage of Ray Kubernetes pod.'
(raylet, ip= Arguments: (FileNotFoundError(2, 'No such file or directory'),)

Docker Desktop - v4.3.2
Kubernetes - v1.22.4
Mac OS - v12.1

Thanks in advance!

Hi @lihost, this is a known issue that’s being tracked here. It seems to be an issue with local K8s setups because another user reported it on minikube.

Hi @shrekris , thank you for the updates.

While this is being resolved, would you suggest any workaround to this so that we can test our deployments before moving to prod?

Turns out there’s another thread solving the same error: Logging error computing CPU usage of Ray Kubernetes Please take a look there!

Yes, I have put in place the work-around - Downgrading Docker Desktop to 4.2.0