# Reporting custom validation and training metrics

**URL:** <https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702>\
**Category:** Ray Tune\
**Created:** [April 13, 2021, 3:01pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702 "2021-04-13T15:01:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gagasth](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gagasth/32/812_2.png) [@gagasth](https://discuss.ray.io/u/gagasth)\
**Post date:** [April 13, 2021, 3:01pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/1 "2021-04-13T15:01:35Z")

</div>

I am new to ray tune, I was able to run a Keras neural network with grid search using ray tune. However, I am unable to report custom metrics. I am interested in report training and validation AUC values apart from the accuracy. I would also like to use AUC as a stopping criteria as tune is overfitting when I use only accuracy. How do I first calculate and then report training and validation AUC ?

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [April 13, 2021, 6:03pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/2 "2021-04-13T18:03:06Z")

</div>

cc @kai Can you address the question?

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [April 13, 2021, 6:14pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/3 "2021-04-13T18:14:17Z")

</div>

Hi @gagasth, can you share the code you’re currently running? Especially the trainable function would be interesting.

Generally you report custom metrics via `tune.report()` in the trainable function.

---

<div class="post-metadata">

**Author:** ![gagasth](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gagasth/32/812_2.png) [@gagasth](https://discuss.ray.io/u/gagasth)\
**Post date:** [April 14, 2021, 7:06pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/4 "2021-04-14T19:06:39Z")

</div>

Hi @kai, thanks for the response. I am looking to stop if (testAUC - trainAUC \> 0.1) I have attached a screenshot of my code. Not seen in the screen shot is the dataloader function and the libraries I have imported.

List of libraries  
ray  
from ray.tune.schedulers import AsynHyperBandScheduler  
from ray.tune. integration.keras import TuneReportCallBack

 ![Screen Shot 2021-04-14 at 12.26.32 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/a95326e03db5f69d3d295e3be36234a36bc52738.jpeg)

---

<div class="post-metadata">

**Author:** ![gagasth](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gagasth/32/812_2.png) [@gagasth](https://discuss.ray.io/u/gagasth)\
**Post date:** [April 20, 2021, 5:36pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/5 "2021-04-20T17:36:37Z")

</div>

@kai and @sangcho : Just checking to see if you had any questions or solutions for me? Thank you.

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [April 20, 2021, 5:53pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/6 "2021-04-20T17:53:15Z")

</div>

Hi @gagasth, I’ve actually looked into this. The problem here seems to be that Keras callbacks either report the training results or the test results, but not both at the same time. Thus the TuneReportCallback might not work for your use case.

You need to get the training and test evaluation scores together. You could potentially use a stateful Keras callback for that.

Basically you’ll need to have `train_auc` and `test_auc` and the do something like `tune.report(train_auc=train_auc, test_auc=test_auc, auc_diff=test_auc-train_auc)`.

For early stopping, you’ll then have to implement a [custom stopper](https://docs.ray.io/en/master/tune/api_docs/stoppers.html) to implement that specific early stopping mechanism. Alternatively you can just stop training within the trainable function - i.e. by setting `self.model.stop_training = True` in the callback.

---

<div class="post-metadata">

**Author:** ![gagasth](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gagasth/32/812_2.png) [@gagasth](https://discuss.ray.io/u/gagasth)\
**Post date:** [April 20, 2021, 6:47pm UTC](https://discuss.ray.io/t/reporting-custom-validation-and-training-metrics/1702/7 "2021-04-20T18:47:52Z")

</div>

Ok. Thank you. I am new to this and an example will be helpful. Would you happen to have an example for using tune.report with stateful keras callback?