# Keras callbacks - training or validation accuracy

**URL:** <https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337>\
**Category:** Ray Tune\
**Created:** [December 29, 2020, 1:25pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337 "2020-12-29T13:25:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [December 29, 2020, 1:25pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/1 "2020-12-29T13:25:13Z")

</div>

Hello  
I wounder about some piece of example code from: [https://docs.ray.io/en/latest/tune/examples/tune\_mnist\_keras.html](https://docs.ray.io/en/latest/tune/examples/tune_mnist_keras.html)

```
model.fit(
    x_train,
    y_train,
    batch_size=batch_size,
    epochs=epochs,
    verbose=0,
    validation_data=(x_test, y_test),
    callbacks=[TuneReportCallback({
        "mean_accuracy": "accuracy"
    })])

```

Is the “mean\_accuracy”: “accuracy” training or validation accuracy? How to change it?  
In typical keras usage are: ‘acc’, ‘val\_acc’, ‘loss’ and ‘val\_loss’.

Regards Peter

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [December 29, 2020, 7:07pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/2 "2020-12-29T19:07:15Z")

</div>

> [@Peter\_Pirog](#):
>
> `accuracy`

I think this might be an example from a different version of Keras (or TF2?). I think you can just change it to:

```auto
TuneReportCallback({"accuracy": "acc"})

```

---

<div class="post-metadata">

**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\
**Post date:** [December 29, 2020, 7:13pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/3 "2020-12-29T19:13:49Z")

</div>

Hi @Peter_Pirog “accuracy” here is referring to training accuracy. This metric is specified in the model.compile(…) line in the example right before we call model.fit(…).

According to the tensorflow docs, passing in either “accuracy” or “acc” to the metrics field in model.compile should have the same effect.

> When you pass the strings ‘accuracy’ or ‘acc’, we convert this to one of[`tf.keras.metrics.BinaryAccuracy`](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/BinaryAccuracy),[`tf.keras.metrics.CategoricalAccuracy`](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/CategoricalAccuracy),[`tf.keras.metrics.SparseCategoricalAccuracy`](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/SparseCategoricalAccuracy) based on the loss function used and the model output shape.

[https://www.tensorflow.org/api\_docs/python/tf/keras/Model#compile](https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile)

---

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [December 29, 2020, 10:16pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/4 "2020-12-29T22:16:49Z")

</div>

Thank You for the answers. Do you think is possible to use validation accuracy as metric in tune.run() function? Thank you for info, how to change type of accuracy metric.

Peter

---

<div class="post-metadata">

**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\
**Post date:** [December 30, 2020, 1:21am UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/5 "2020-12-30T01:21:50Z")

</div>

I believe Keras automatically tracks validation metrics by just pretending “val” to the metric name. So you would first have to specify “val\_accuracy” or “val\_acc” in the TuneReportCallback that gets passed into model.fit: `TuneReportCallback(“val_accuracy”)`.

This will cause the validation accuracy to get reported to Tune. And then when you call tune.run, specify this as the metric argument: `tune.run(..., metric=“val_accuracy”)`.

---

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [December 30, 2020, 6:34am UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/6 "2020-12-30T06:34:56Z")

</div>

Thank you for suggestion. I willa try it.

Peter

śr., 30 gru 2020, 02:32 użytkownik Amog Kamsetty via Ray \<[ray@discoursemail.com](mailto:ray@discoursemail.com)\> napisał:

---

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [December 31, 2020, 9:20am UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/7 "2020-12-31T09:20:58Z")

</div>

@amogkam your suggesstion works fine. I improve my code to:

```auto
    model.fit(
        x_train,
        y_train,
        batch_size=batch_size,
        epochs=epochs,
        verbose=0,
        validation_data=(x_test, y_test),
        callbacks=[TuneReportCallback({
            "mean_accuracy": "val_accuracy" #optional values ['loss', 'accuracy', 'val_loss', 'val_accuracy']
        })])
```

---

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [May 31, 2021, 5:36pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/8 "2021-05-31T17:36:15Z")

</div>

@amogkam @rliaw Now I try to define own custom metris and add it to the tuning.

I defined function:

> ```
> def rmsle(y_pred, y_test) :
> return tf.math.sqrt(tf.reduce_mean((tf.math.log1p(y_pred) - tf.math.log1p(y_test))**2))
> 
> ```

model compile:

> ```
> model.compile(
> loss=rmsle,#mean_squared_logarithmic_error "mse"
> optimizer=tf.keras.optimizers.Adam(lr=config["lr"]),
> metrics=['error',rmsle])
> 
> ```

fit model:

> model.fit(  
> X\_train,  
> y\_train,  
> batch\_size=batch\_size,  
> epochs=epochs,  
> verbose=0,  
> validation\_data=(X\_test, y\_test),  
> callbacks=[TuneReportCallback({  
> “error”: “val\_error”  
> })])  
> and  
> analysis = tune.run(  
> …  
> metric=“error”,…

the error is:

> ```
> report_dict[key] = logs[metric]
> 
> ```
> 
> KeyError: ‘val\_error’

I will be grateful for any suggestions.

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [May 31, 2021, 11:33pm UTC](https://discuss.ray.io/t/keras-callbacks-training-or-validation-accuracy/337/9 "2021-05-31T23:33:02Z")

</div>

Maybe try:

```auto
callbacks=[TuneReportCallback({
“error”: “error”
})])

```

?