How to persist logs directory after head node restart

Monitoring & Debugging
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

By default, Ray log files are stored in a /tmp/ray/session_*/logs directory. (Configuring Logging — Ray 3.0.0.dev0)

I am changing this behaviour using RAY_TMPDIR env variable. For logs persistence, I have mounted a PVC at this location and I want to persist the logs even after an accidental restart of the head node.

But whenever I delete the head node (kubernetes pod), the logs location is cleared from the pvc as well. The Ray dashboard is reset to zero as if no jobs were submitted on the ray-cluster. The new head pod has a different directory for storing the logs now.

Is this possible to persist the logs in the Ray dashboard across a head node restart. I am already using the log persistence (Log Persistence — Ray 3.0.0.dev0) to redirect the logs to a PLG setup. But I want to be able to view the logs in the dashboard as well. How can I achieve this.

2

Persisting logs in the dashboard is currently not on our roadmap, but please file a request on Github

3

Thanks @nikitavemuri for the update. I will file a request on Github.

Could you please also address another query at Log Rotation and Retention Period ? We would want to delete the log files after a certain period in our raycluster to avoid disk space exhaustion.

Thanks