How to Stop Ray based on python condition or bug in code?

Dashboard, Monitoring & Debugging
1
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

What I’m doing: Iterating through thousands of files and processing them using @serve.deployment() on top of my class.

I’m new to Ray, but my question is, how do I stop the ray completely to a standstill (only the nodes that are running the process, since I have other programs running on ray) with a simple python code based on some condition?
Just like how “raise SystemExit()” exits the full python program.
Currently: Even if there are errors in my program (bug), the ray node finishes processing all my files and then I see that all of the files have errors.

EDIT: ray.shutdown() will mess with other programs running on ray, so I just want to stop ray from processing my files if a condition doesnt pass. Can I use raise SystemExit() to stop the processing and is it a good practice?

2

cc: @sangcho and @Sihan_Wang for thoughts.

3

Currently: Even if there are errors in my program (bug), the ray node finishes processing all my files and then I see that all of the files have errors.

Can you show me code example of this?

4

Hello @salman-moh

According to me to stop Ray from processing files based on a condition without affecting other programs you can Use conditional statements within your processing loop to break out early if the condition is met. Compared to raise SystemExit(), which abruptly ends the script, this method offers more control.

Example:`import ray

for file in files_to_process:
if condition_not_met:
break # Exit the loop based on condition

# Process the file using @ray.serve.deployment()
result = ray.get(process_file.remote(file))

`
With this technique, you can gently end Ray’s operations while maintaining compatibility for other Ray-running applications.