Log Rotation and Retention Period

Monitoring & Debugging
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi,

I am using Ray 2.10.0

https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html
According to the above doc - Ray supports log rotation of log files. Note that not all components support log rotation. (Raylet, Python, and Java worker logs do not rotate).

I have set the following env variables in my raycluster.
RAY_ROTATION_MAX_BYTES=1024
RAY_ROTATION_BACKUP_COUNT=1
I am able to see the logs rotation for python-core-driver, python-core-worker

-rw-r–r-- 1 root root 1016 May 14 05:12 python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff_5951.1.log
-rw-r–r-- 1 root root 1.0K May 14 05:12 python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff_5951.log
-rw-r–r-- 1 root root 983 May 14 05:12 python-core-worker-5c474bde48bb7042a2f4ef2ce3518e670cf1cd38434946950635fdf8_5862.1.log
-rw-r–r-- 1 root root 962 May 14 05:12 python-core-worker-5c474bde48bb7042a2f4ef2ce3518e670cf1cd38434946950635fdf8_5862.log

Does this suggest that log rotation is now available for python worker logs ?

But the raylet logs are still not rotating as the file size has already exceeded the limit set by the env variable RAY_ROTATION_MAX_BYTES.
-rw-r–r-- 1 root root 239K May 14 05:15 raylet.out

Does this mean that Raylet logs still do not support log rotation ?

job-driver-[submission_id].log: The stdout of a job submitted with the Ray Jobs API.

Is there a way to delete the logs which are older than a certain number of days. (7 for example)

I want to avoid errors in my raycluster due to disk space exhaustion.
Any assistance would be highly appreciated. Thanks.

2

@Huaiwei_Sun kindly assist to understand the current behaviour and limitations (if any) to help us achieve the Log rotation and retention as described above. Thanks.

3

Hi @ayushraj

I am able to see the logs rotation for python-core-driver, python-core-worker
-rw-r–r-- 1 root root 1016 May 14 05:12 python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff_5951.1.log
-rw-r–r-- 1 root root 1.0K May 14 05:12 python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff_5951.log
-rw-r–r-- 1 root root 983 May 14 05:12 python-core-worker-5c474bde48bb7042a2f4ef2ce3518e670cf1cd38434946950635fdf8_5862.1.log
-rw-r–r-- 1 root root 962 May 14 05:12 python-core-worker-5c474bde48bb7042a2f4ef2ce3518e670cf1cd38434946950635fdf8_5862.log

Q: Does this suggest that log rotation is now available for python worker logs ?

A: These are system logs. Not the “worker logs” that users see.

Q: Does this mean that Raylet logs still do not support log rotation ?

A: The doc serves as the source of truth.
https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-rotation

Q: Is there a way to delete the logs which are older than a certain number of days. (7 for example)
A: There are a couple of tools to do it. We don’t have official answer or recommendation for this.

2 Likes