Hi, I am using Ray Tune to do hyperparameter tuning of a Pytorch model. The tuning uses cross validation. After each fold, I report the loss value, and in case this is infinite/nan, I would like to cancel the trial and any further fold. Currently, I have set up a stopping criteria
def stop_fn(trial_id, result):
return not np.isfinite(result["score"])
but if this is triggered, the trial will be Terminated rather than Cancelled. This means that, first, I would not see if the trial ended with a not-define loss in the general report, and second, the partial score of the trial (i.e., not for all folds) will be taken into account to retrieve the best model at the end.
So, it is possible to cancel a trial rather than terminate/stop it?