How to force ray tune to shutdown from inside to continue experiment later

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

If i raise Exceptions during training, Ray tune does not seem to stop, even if I set max_failures=0. Thus, I want to forcefully stop ray experiment from the inside to be able to resume experiment later.

I read in User Guide & Configuring Tune — Ray v1.10.0 that one must send a SIGINT signal to the ray main worker to shutdown.

What is the best practice to do this from within a worker? Assumming i have PBT Training and many workers, i cannot do it from the outside.

2

Hey @wfskmoney, max_failures=0 means that if a trial fails, that particular trial won’t be retried, but all the other trials will still be executed.

If you want to stop the entire experiment when any trial fails, you can set fail_fast=True in tune.run. Let me know if this works for you!