How severe does this issue affect your experience of using Ray?
I have 3 custom metrics implemented using Callbacks. For my use case, the best models score well on all 3 metrics for both train and test (these are much more important than the ‘reward’). So in total it’s 6 metrics for me to keep track of in Tensorboard. It’s also impossible to use
checkpoint_score_attr in this way.
I have come up with a formula which takes all 6 metrics as input and produces a single score. This score directly represents how good the model is.
It Is it possible to create a custom metric which can have both train and evaluation results as input?
Right now I’m using a callback to generate the custom metrics, but it seems to me that during each callback I only have access to either train or evaluation.