Optimize a nested metric

Ray Tune
1

I am working with rllib and tune but met some difficulties.
I want to optimize a metric reported only by my customized evaluation, but all of the metrics reported by evaluation are nested in the reported result. So I can’t pass such a metric into tune.run() since it is not a key of reported result. Similarly, I can’t pass such a metric as ‘checkpoint_score_attr’ into tune.run() for the same reason.
Is there any way to do the same thing as I want? Or may be we can add this new feature?

2

Hi Raphael, right, currently this is not supported. Can you instead pass the metric directly to tune.report? Like this

result = evaluate(...)
tune.report(metric=result["nested"]["metric"], **result)

and then pass metric=metric to tune.run?

3

Oh sorry, I just saw you’re using RLLib. In that case cc @sven1977 who might know where to do this best.

4

Thanks for mentioning him!

5

I pull request here

6

That’s awesome, thanks again!

1 Like