How to Stop Ray based on python condition or bug in code?

Monitoring & Debugging
1
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

What I’m doing: Iterating through thousands of files and processing them using @serve.deployment() on top of my class.

I’m new to Ray, but my question is, how do I stop the ray completely to a standstill (only the nodes that are running the process, since I have other programs running on ray) with a simple python code based on some condition?
Just like how “raise SystemExit()” exits the full python program.
Currently: Even if there are errors in my program (bug), the ray node finishes processing all my files and then I see that all of the files have errors.

EDIT: ray.shutdown() will mess with other programs running on ray, so I just want to stop ray from processing my files if a condition doesnt pass. Can I use raise SystemExit() to stop the processing and is it a good practice?

2

cc: @sangcho and @Sihan_Wang for thoughts.

3

Currently: Even if there are errors in my program (bug), the ray node finishes processing all my files and then I see that all of the files have errors.

Can you show me code example of this?