Hi! I get this raylet message every 4-5 secs and it’s annoying. How do I disable it or increase the time these messages show up?
“(raylet) [2022-11-30 07:51:47,265 E 1805 1852] (raylet) file_system_monitor.cc:105: /tmp/ray/session_2022-11-30_05-45-20_473636_1539 is over 95% full, available space: 856064; capacity: 8132173824. Object creation will fail if spilling is required.”
If I want to log these raylet messages in a log file instead of showing them in the terminal, how can I do that?
Unfortunately there is no way to disable logging for only this log message, but you could try setting log_to_driver=False in ray.init(). Note, though, that this will turn off all logging.
For now, I’ve opened a couple issues to make this better in the future, here and here.
@Stephanie_Wang I see we cannot disable them, it will disable all if do. But is there any way that instead of logging it in the terminal, it logs into a file? So that terminal does not gets full. And all the update that happens in the terminal log messages, the same thing happens to the log file the same way like updating the results from tune.Tuner method?
You could try a bash command to write both stderr and stdout (or only one of these) directly to a file like this:
$ python test.py 2>&1 > out