How to detect when a rayTune experiment has been interrupted by a KeyboardInterrupt (CTRL+C)

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hello,

I have a difficulty to detect when a ray tune experiment got interrupted by a KeyboardInterrupt (CTRL+C).
In my usecase, I only want to look at the best model and evaluate its performance on the test set in case I got a fully successful and completed ray tune experiment. For now I’m checking on the results.num_errors (results=tuner.fit()) field and I have validated that if I for instance raise an exception from my trainable, then this variable is incremented. However, when I do a single CTRL+C to interrupt the trainable, then this variable remains at 0. I cannot figure out how to identify that my experiment didn’t fully execute. Any advice on how to detect that condition ?

side note: In my trainable, I’m using TensorFlow keras to do the training of the model, don’t know if this has any impact or not on the problem…

Thanks,

2

For anybody who would be interested, I found a way to do that, not sure this is the best way or not but anyway…
What I do to detect this is to run through all the result objects and check if any has its result.metrics[“done”] field set to False.