You are correct: not all Ray log files are rotated, which can lead to large log files and /tmp filling up, especially in multi-user environments. As of recent Ray versions, only some logs (like python-core-worker logs) support rotation, while others (notably raylet, worker, and GCS server .out/.err files) do not, though work is ongoing to add rotation for these as well. You can configure log rotation for supported components using the environment variables RAY_ROTATION_MAX_BYTES and RAY_ROTATION_BACKUP_COUNT, but this will not affect all log types yet. Full log rotation for all components is a requested and actively developed feature, with improvements expected in upcoming releases (docs, GitHub issue, PR).
For now, you may need to manually monitor and clean up large log files, or use external tools/scripts for log management. There is no official Ray mechanism yet for deleting old logs by age, and some log types (especially .err files) are still not rotated (Discourse). Would you like more detail on which log files are currently rotated and how to configure rotation for those that are?
Sources:
- Ray GitHub issue: Log rotation on workers
- Ray PR: Fix gcs and raylet logging for stdout
- Discourse: How to rotate .err log files?
Hint: Mention @RunLLM in the post for followups.