Ray is creating hundreds of logs files under /tmp/ray/session_latest/logs/ causing disk space issue and I/O Spikes

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Ray v2.6.3, Ray is creating hundreds of files under /tmp/ray/session_latest/logs/ worker-*.err, worker-*.out and python-core-worker-*.logs.
I would like to understand how do I control the Event stats log printed by ray. Also I could see lot of noise in the logs due to RAY_LOG(INFO).

How do I completely disable these logs ? and also stop ray from creating so many log files, It’s causing a lot of IO spikes which I want to avoid. Could someone please help me asap ?

2

Are you still seeing this issue in Ray 2.7.1? We’re adding logging improvements to future versions of Ray, so those will also help with your issue.

3

I have the same problem. My Python program uses ray version 2.7.1 and runs in a Docker container. Ray creates many files under /tmp/ray/session_latest/logs/python-core-worker-.log, worker-.err and worker-*.out logs. Which eat up all my disk space over time. When the disk space is completely eaten up, the Docker container simply stops working. If I restart the container, it works again, but this is only a workaround and not a long-term solution for me.

I tried to control the logs using the ray documentation (Configuring Logging — Ray 2.9.3):

  1. "Disable logging to the driver by setting “ray.init(log_to_driver=False)” in the ray initialization. This will only cause the logs to not show up in the container logs, but they will still be logged in the above files. So this is not a solution.
  2. Configure “log rotation” by setting environmental variables in the docker-compose.yml file:
    “RAY_ROTATION_MAX_BYTES=1024”
    “RAY_ROTATION_BACKUP_COUNT=1”
    But this doesn’t change anything. Even setting the environment variables to ‘0’ has no effect.
    I think this is because the Python components don’t support log rotation (“Note that not all components support log rotation. (Raylet, Python, and Java worker logs do not rotate)”).

Is there a way to avoid these Raylet log files or control their number and size using Python?
Is there a way to show only the ray logs in the container logs without saving them to files?

Any help would be greatly appreciated!
Thanks in advance!

4

Hello @shrekris how do you control the logs using improved logging in Ray 2.7.1? I already tried these things mentioned here but without success. I’m running out of ideas…
Any help would be greatly appreciated!
Thanks in advance!

1 Like
5

I am also running into the same issue on ray 2.9.1. There are a ton of large worker logs.

6

[Using Ray core] Facing same problem here, after long run on sever cluster, the logs taking up all spaces… Would be good to have an option to handle that.

7

There’s a env variable which controls the threshold for logging: ray/src/ray/util/logging.cc at b63a4ca22df67247eeaafaa3ecb6b888b5cb8bab · ray-project/ray · GitHub.
My understanding is, if you upgrade the threshold to, say, error, a lot of the logging message won’t be printed.

8

Configuring log rotation by RAY_ROTATION_MAX_BYTES and RAY_ROTATION_BACKUP_COUNT is also feasible to control the max size of logging files.