Feature request: [Tune] CLIReporter: sort and show top results in the intermediate table

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Can CLIReporter show the top results sorted by metric? If we have a lot of experiments, we interest more on the top results.

Thanks!

1 Like
2

Hi @Eleven1Liu, currently this is not possible. At the moment we only display the top result right above the table.

Can you share why you want to take a look at more than one top result? Generally it should be possible and not that hard to implement, I’d just like to understand the use case better.

3

Hi @kai,

Great!
The first reason is that the top one includes too much info (all parameters). We hope to have a table summarizing top metrics.

Another reason could be to better understand the effect of hyperparameters. A sorted table will allow us to know the impact of parameters more clearly.

4

Sounds like a good idea to add this as an option to the CLIReporter.

I was wondering if you would be willing to try to contribute this to Ray Tune yourself (i.e. submitting a PR) - I’d be happy to guide you through this. Is this something you’d like to try?

5

Maybe we can go ahead and file this request on github? posted here [tune] Sort top results by metric · Issue #16448 · ray-project/ray · GitHub

6

@kai Great!! :smiley: Should I start from submitting a PR?

7

You can just start! A couple of pointers:

  1. We should make sure that ordering by metric is optional. The default should still be like it is today
  2. The option should make use of the metric and mode passed to tune.run(), if not specified directly
  3. It would be great if you could add a test for this as well.

If you need more guidance, please feel free to ping me. Also, if you open a PR, you can assign it to me (or ping my github account @krfricke)

2 Likes
8

Any updates here? Would love to see the top results sorted