Hello,
I am trying to define a custom metric function using ray tune. Currently, the neural network is trained for many epochs and stops once the validation loss starts increasing. So by default the ASHA scheduler takes the minimum validation at the end to compare the models with different hyperparameters. But what I want to do is to run the neural network(NN) for fixed number of epochs, say 10000, and pick the one that minimizes validation within this number of epochs. The problem is that some models show increasing validation loss at low epochs and decreases drastically over epochs. But I am not sure how to get started. Any ideas would be helpful.
Thank you.