Using tune.run(), I’m trying to save some checkpoints with “good” custom_metric values. The custom_metric value is computed via a custom callback only during evaluation. Reading the tune.run docs, it seems that I could keep the last keep_checkpoints_num best checkpoints using checkpoint_score_attr as score.
If that is true, how can I use my custom metric as score?
tune.run( keep_checkpoints_num=3, checkpoint_score_attr="evaluation/custom_metrics/score" )
Btw, this score metric shows up in tensorboard as
If lower score is better, do I write:
tune.run( keep_checkpoints_num=3, checkpoint_score_attr="min-evaluation/custom_metrics/score" )
tune.run( keep_checkpoints_num=3, checkpoint_score_attr="evaluation/custom_metrics/score", mode='min', )
My tune.run() looks like this:
tune.run( "A3C", name="study", config=config, stop=stop, )