Disk Pressure due to accumulating logs and runtime_resources

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

we’re having a Ray cluster setup in Kubernetes and we noticed disks getting full over time with logs and runtime resources. As a result our head node would flip with disk_pressure error once it reached 91% of disk utilization. I’ve created a cron task to clear files from /tmp/ray location for previous sessions, preserving only what’s in session_latest.
What are the consequences of deleting those files? How can we be sure we won’t disrupt an user by deleting the runtime_env that he’s using? What is the preffered way to handle this situation?

Please refer to the screenshot