Ray is creating hundreds of logs files under /tmp/ray/session_latest/logs/ causing disk space issue and I/O Spikes

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
3

I have the same problem. My Python program uses ray version 2.7.1 and runs in a Docker container. Ray creates many files under /tmp/ray/session_latest/logs/python-core-worker-.log, worker-.err and worker-*.out logs. Which eat up all my disk space over time. When the disk space is completely eaten up, the Docker container simply stops working. If I restart the container, it works again, but this is only a workaround and not a long-term solution for me.

I tried to control the logs using the ray documentation (Configuring Logging — Ray 2.9.3):

  1. "Disable logging to the driver by setting “ray.init(log_to_driver=False)” in the ray initialization. This will only cause the logs to not show up in the container logs, but they will still be logged in the above files. So this is not a solution.
  2. Configure “log rotation” by setting environmental variables in the docker-compose.yml file:
    “RAY_ROTATION_MAX_BYTES=1024”
    “RAY_ROTATION_BACKUP_COUNT=1”
    But this doesn’t change anything. Even setting the environment variables to ‘0’ has no effect.
    I think this is because the Python components don’t support log rotation (“Note that not all components support log rotation. (Raylet, Python, and Java worker logs do not rotate)”).

Is there a way to avoid these Raylet log files or control their number and size using Python?
Is there a way to show only the ray logs in the container logs without saving them to files?

Any help would be greatly appreciated!
Thanks in advance!