# Custom metric that combines train and eval results

**URL:** <https://discuss.ray.io/t/custom-metric-that-combines-train-and-eval-results/6450>\
**Category:** RLlib\
**Created:** [June 9, 2022, 7:36pm UTC](https://discuss.ray.io/t/custom-metric-that-combines-train-and-eval-results/6450 "2022-06-09T19:36:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ihopethiswillfi](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@ihopethiswillfi](https://discuss.ray.io/u/ihopethiswillfi)\
**Post date:** [June 9, 2022, 7:36pm UTC](https://discuss.ray.io/t/custom-metric-that-combines-train-and-eval-results/6450/1 "2022-06-09T19:36:36Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium.

I have 3 custom metrics implemented using Callbacks. For my use case, the best models score well on all 3 metrics for **both train and test** (these are much more important than the ‘reward’). So in total it’s 6 metrics for me to keep track of in Tensorboard. It’s also impossible to use `checkpoint_score_attr` in this way.

I have come up with a formula which takes all 6 metrics as input and produces a single score. This score directly represents how good the model is.

It Is it possible to create a custom metric which can have both train and evaluation results as input?

Right now I’m using a callback to generate the custom metrics, but it seems to me that during each callback I only have access to either train or evaluation.

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [June 10, 2022, 3:36pm UTC](https://discuss.ray.io/t/custom-metric-that-combines-train-and-eval-results/6450/2 "2022-06-10T15:36:21Z")

</div>

Hi @ihopethiswillfi ,

The metrics of your evaluation and training are simply thrown into the same dictionary and handed back to tune.  
There is no callback on RLlib’s side that has access to both.  
But if you don’t evaluate in parallel to training, but afterwards, you can include your training metric in the evaluation metric if that helps.

I’m assuming here, that you are talking about evaluation metrics, when you say “_metrics for […] test_”, correct?

---

<div class="post-metadata">

**Author:** ![ihopethiswillfi](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@ihopethiswillfi](https://discuss.ray.io/u/ihopethiswillfi)\
**Post date:** [June 10, 2022, 8:25pm UTC](https://discuss.ray.io/t/custom-metric-that-combines-train-and-eval-results/6450/3 "2022-06-10T20:25:32Z")

</div>

Hi @arturn

Yes, my evaluation metrics are calculated on test.

I did manage to calculate the score (which combines both train and eval metrics) after training, and also during a CustomStopper function.

However, it seems impossible for things like `checkpoint_score_attr`.

Problem not solved, but thanks for confirming it’s impossible.