Ray is creating hundreds of logs files under /tmp/ray/session_latest/logs/ causing disk space issue and I/O Spikes

Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Ray v2.6.3, Ray is creating hundreds of files under /tmp/ray/session_latest/logs/ worker-*.err, worker-*.out and python-core-worker-*.logs.
I would like to understand how do I control the Event stats log printed by ray. Also I could see lot of noise in the logs due to RAY_LOG(INFO).

How do I completely disable these logs ? and also stop ray from creating so many log files, It’s causing a lot of IO spikes which I want to avoid. Could someone please help me asap ?

2

Are you still seeing this issue in Ray 2.7.1? We’re adding logging improvements to future versions of Ray, so those will also help with your issue.

3

I have the same problem. My Python program uses ray version 2.7.1 and runs in a Docker container. Ray creates many files under /tmp/ray/session_latest/logs/python-core-worker-.log, worker-.err and worker-*.out logs. Which eat up all my disk space over time. When the disk space is completely eaten up, the Docker container simply stops working. If I restart the container, it works again, but this is only a workaround and not a long-term solution for me.

I tried to control the logs using the ray documentation (Configuring Logging — Ray 2.9.3):

  1. "Disable logging to the driver by setting “ray.init(log_to_driver=False)” in the ray initialization. This will only cause the logs to not show up in the container logs, but they will still be logged in the above files. So this is not a solution.
  2. Configure “log rotation” by setting environmental variables in the docker-compose.yml file:
    “RAY_ROTATION_MAX_BYTES=1024”
    “RAY_ROTATION_BACKUP_COUNT=1”
    But this doesn’t change anything. Even setting the environment variables to ‘0’ has no effect.
    I think this is because the Python components don’t support log rotation (“Note that not all components support log rotation. (Raylet, Python, and Java worker logs do not rotate)”).

Is there a way to avoid these Raylet log files or control their number and size using Python?
Is there a way to show only the ray logs in the container logs without saving them to files?

Any help would be greatly appreciated!
Thanks in advance!

4

Hello @shrekris how do you control the logs using improved logging in Ray 2.7.1? I already tried these things mentioned here but without success. I’m running out of ideas…
Any help would be greatly appreciated!
Thanks in advance!

1 Like
5

I am also running into the same issue on ray 2.9.1. There are a ton of large worker logs.

6

[Using Ray core] Facing same problem here, after long run on sever cluster, the logs taking up all spaces… Would be good to have an option to handle that.

7

There’s a env variable which controls the threshold for logging: ray/src/ray/util/logging.cc at b63a4ca22df67247eeaafaa3ecb6b888b5cb8bab · ray-project/ray · GitHub.
My understanding is, if you upgrade the threshold to, say, error, a lot of the logging message won’t be printed.

8

Configuring log rotation by RAY_ROTATION_MAX_BYTES and RAY_ROTATION_BACKUP_COUNT is also feasible to control the max size of logging files.

9

I’m facing the same issue:

My application uses Ray 2.46.0 inside a Docker container. Even after setting these environment variables:

ENV RAY_BACKEND_LOG_LEVEL=error
ENV RAY_LOG_TO_DRIVER_EVENT_LEVEL=error
ENV LOGGER_LEVEL=error

Ray still creates a large number of worker-*.out and worker-*.err files. These files only contain the task name being executed, which generates a lot of noise and consumes disk space.

Is there a way to completely disable this logging — i.e. no log files created at all, and no such messages printed to stdout?

1 Like
10

A problem for us too. Users started to complain about warnings of /tmp being full and there were 600MB to 1.4GB logs. As I have understood the problematic files are the ones that are not being rotated by ray yet.

This wouldn’t be a problem with one user, but we have multiple users on the same machine each of which have their own ray environment. Getting everything rotated would be really nice.

11

You are correct: not all Ray log files are rotated, which can lead to large log files and /tmp filling up, especially in multi-user environments. As of recent Ray versions, only some logs (like python-core-worker logs) support rotation, while others (notably raylet, worker, and GCS server .out/.err files) do not, though work is ongoing to add rotation for these as well. You can configure log rotation for supported components using the environment variables RAY_ROTATION_MAX_BYTES and RAY_ROTATION_BACKUP_COUNT, but this will not affect all log types yet. Full log rotation for all components is a requested and actively developed feature, with improvements expected in upcoming releases (docs, GitHub issue, PR).

For now, you may need to manually monitor and clean up large log files, or use external tools/scripts for log management. There is no official Ray mechanism yet for deleting old logs by age, and some log types (especially .err files) are still not rotated (Discourse). Would you like more detail on which log files are currently rotated and how to configure rotation for those that are?

Sources:

Hint: Mention @RunLLM in the post for followups.