Why Raylet and Python/Java worker logs are not rotating

Monitoring & Debugging
1

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity
  • Low: It annoys or frustrates me for a moment.
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.
  • High: It blocks me to complete my task.

Accroding to the Ray Docs,

Raylet and Python/Java worker logs are not rotating

my python ray worker has a lot logs. How can I make logs of worker rotate?
My tmp workaround is set env RAY_LOG_TO_STDERR=1 , disable all ray log

2

cc: @sangcho who is working on logging.

3

Sorry to bother again, is there anyon who works on log rotation?
cc @sangcho

4

@sangcho works on it but he is out on vacation for now. He’ll get back to you after he is back next week

1 Like
5

The best way to achieve this now is to use your own logger that has log rotate handler!

6

@sangcho I believe we should solve this issue totally, instead of taking it over to users.
This is not the first time we met it.

If you guys don’t have bandwidth on it, I could take it to our team.

1 Like
7

We are planning to tackle this in ray 2.5. I am open to discuss the design btw.

1 Like