How severe does this issue affect your experience of using Ray?
- High: It blocks me to complete my task.
I have a difficulty to detect when a ray tune experiment got interrupted by a KeyboardInterrupt (CTRL+C).
In my usecase, I only want to look at the best model and evaluate its performance on the test set in case I got a fully successful and completed ray tune experiment. For now I’m checking on the results.num_errors (results=tuner.fit()) field and I have validated that if I for instance raise an exception from my trainable, then this variable is incremented. However, when I do a single CTRL+C to interrupt the trainable, then this variable remains at 0. I cannot figure out how to identify that my experiment didn’t fully execute. Any advice on how to detect that condition ?
side note: In my trainable, I’m using TensorFlow keras to do the training of the model, don’t know if this has any impact or not on the problem…