How severe does this issue affect your experience of using Ray?
Medium: It contributes to significant difficulty to complete my task, but I can work around it.
We have a pipeline that is a mixture of python code running directly in Ray, and some CLI apps that we run via python’s subprocess module. These CLI apps can use significant amounts of memory, and we’d like them to be considered by Ray’s memory management - both for OOM killing and metrics.
Am I right in thinking this does not happen right now? I’m pretty sure based on the behavior I’m seeing that it’s is true, but wanted to check. Is there any way I can trick Ray into tracking a child process’ memory as if it’s part of the parent?
I found the same problem, and Ray officials do not seem to recommend using subprocess to create processes. But I think Ray officially can easily obtain the sub-process under Ray Task by calling the GetAllProcsWithPpid method in the source code and manage it when appropriate.
I found that python parallel and distributed computing is made possible with the well-liked Ray framework for creating distributed applications. The capacity to control and monitor the resources used by tasks and actors, including memory use, is one of its strong points. The user experience can be greatly impacted by memory monitoring of child processes in Ray, both positively and negatively.