How severe does this issue affect your experience of using Ray?
- Low: It annoys or frustrates me for a moment.
Hello everyone,
When using Ray, some of my worker processes encounter segmentation faults which leads to their death. This is not a problem and it is expected for my usecase.
However, I would like to supress these logging messages that I get from raylet:
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 7608d962d6e863f7e256255e2d00316fe3e5fe2401000000 Worker ID: 7994c475dd35dba0050cb000bced5bce70328d0149ad02a79d78b88f Node ID: 3cf007159321fe7b43c2c0b4a3f4596d84e8b1b3c4f43cf72a448feb Worker IP address: 193.136.178.219 Worker port: 46315 Worker PID: 3906255 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
How can I do so?