We have an application that uses ray cluster with two servers to distribute work. It is supposed to process 20-100 files. This application spawns worker processes on these two servers/nodes. We want to write log statements to a separate file (apart from ray logging) which we are indexing for splunk log. we are using logger module.
If the application is run on server 1, all the logs (we define the logs in the process, like processing the file X, calling some third party application is successful, etc…) are to be written to server 1, some file let’s say splspyaudit.log in some location.
same like if the application is run on server 2 , all the logs are supposed to be written to server 2 , which also has the same file in the same location on that server.
But the problem here is, we are missing some logs. some of the worker processes are not being able to write to this log file. but we can see that all the worker processes ran successfully, can see the logs under /tmp/ray/session_latest/logs/worker-xxxx.out.
is there a way to make all the worker processes to write to a single file?