Custom metric that combines train and eval results

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Medium.

I have 3 custom metrics implemented using Callbacks. For my use case, the best models score well on all 3 metrics for both train and test (these are much more important than the ‘reward’). So in total it’s 6 metrics for me to keep track of in Tensorboard. It’s also impossible to use checkpoint_score_attr in this way.

I have come up with a formula which takes all 6 metrics as input and produces a single score. This score directly represents how good the model is.

It Is it possible to create a custom metric which can have both train and evaluation results as input?

Right now I’m using a callback to generate the custom metrics, but it seems to me that during each callback I only have access to either train or evaluation.

2

Hi @ihopethiswillfi ,

The metrics of your evaluation and training are simply thrown into the same dictionary and handed back to tune.
There is no callback on RLlib’s side that has access to both.
But if you don’t evaluate in parallel to training, but afterwards, you can include your training metric in the evaluation metric if that helps.

I’m assuming here, that you are talking about evaluation metrics, when you say “metrics for […] test”, correct?

3

Hi @arturn

Yes, my evaluation metrics are calculated on test.

I did manage to calculate the score (which combines both train and eval metrics) after training, and also during a CustomStopper function.

However, it seems impossible for things like checkpoint_score_attr.

Problem not solved, but thanks for confirming it’s impossible.