How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
Hello,
we’re having a Ray cluster setup in Kubernetes and we noticed disks getting full over time with logs and runtime resources. As a result our head node would flip with disk_pressure error once it reached 91% of disk utilization. I’ve created a cron task to clear files from /tmp/ray location for previous sessions, preserving only what’s in session_latest.
What are the consequences of deleting those files? How can we be sure we won’t disrupt an user by deleting the runtime_env that he’s using? What is the preffered way to handle this situation?
Thanks!
Please refer to the screenshot
