Ray lost /prof/stat

I run some Rllib code in singularity on an HPC. For some reason, I need to run 4 different tasks in a node, each with 1 GPU only, and each has its own Ray in singularity image.

Ray version is 2.2
singularity is 3.7

however, it lost /prof/stat on everyday 15pm to 17pm

exception in thread Thread-23:
Traceback (most recent call last):
File “-/.conda/envs/hpc/lib/python3.9/threading.py”, line 980, in _bootstrap_inner
self.run()
File “-/envs/hpc/lib/python3.9/site-packages/ray/tune/utils/util.py”, line 118, in run
self._read_utilization()
File “-/.conda/envs/hpc/lib/python3.9/site-packages/ray/tune/utils/util.py”, line 86, in _read_utilization
float(psutil.cpu_percent(interval=None))
File “-/.conda/envs/hpc/lib/python3.9/site-packages/ray/thirdparty_files/psutil/init.py”, line 1752, in cpu_percent
_last_cpu_times = cpu_times()
File “-/.conda/envs/hpc/lib/python3.9/site-packages/ray/thirdparty_files/psutil/init.py”, line 1613, in cpu_times
return _psplatform.cpu_times()
File “-/.conda/envs/hpc/lib/python3.9/site-packages/ray/thirdparty_files/psutil/_pslinux.py”, line 586, in cpu_times
with open_binary(‘%s/stat’ % procfs_path) as f:
File “-/.conda/envs/hpc/lib/python3.9/site-packages/ray/thirdparty_files/psutil/_common.py”, line 728, in open_binary
return open(fname, “rb”, buffering=FILE_READ_BUFFER_SIZE)

FileNotFoundError: [Errno 2] No such file or directory: ‘/proc/stat’
2%|▏ | 249/9999 [18:30:59<710:30:05, 262.34s/it]2023-07-26 15:55:47,809 ERROR actor_manager.py:486 – Ray error, taking actor 1 out of service. The actor died unexpectedly before finishing this task.
class_name: RolloutWorker
actor_id: ed9e08776c866e5e3c9dde8701000000
pid: 24807