Ray Tune confidence interval

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hi I am currently trying to do hyper parameter optimization with ray tune.

I want to find the best learing rate for some objective function.

However I want to consider the mean and variance of learning rate performance by running it multiple times. Are there any options for mean and variance in ray tune?

2

Hi @limaries30, Tune does not have any numerical analysis tools - but you should be able to easily leverage numpy or pandas for that. E.g. you can use

analysis = tune.run(...)

df = analysis.dataframe()

To get the trial results table as a pandas dataframe.

If you want to run the same config multiple times, you can e.g. use a grid search parameter, like this:

from ray import tune
import numpy as np

def train(config):
    print(config["repeat"], config["a"])
    return np.random.randn() * 0.1 + config["a"]


searcher = tune.suggest.basic_variant.BasicVariantGenerator(
    constant_grid_search=True
)

analysis = tune.run(
    train,
    config={
        "repeat": tune.grid_search(list(range(4))),
        "a": tune.uniform(0, 1)
    },
    search_alg=searcher,
    num_samples=2)

df = analysis.dataframe()

print(df.groupby(["config/a"]).mean())
print(df.groupby(["config/a"]).std())

Output:

== Status ==
Memory usage on this node: 11.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.56 GiB heap, 0.0/2.28 GiB objects
Result logdir: /Users/kai/ray_results/train_2021-07-27_11-15-56
Number of trials: 8/8 (8 TERMINATED)
+-------------------+------------+-------+----------+----------+--------+------------------+-----------+
| Trial name        | status     | loc   |        a |   repeat |   iter |   total time (s) |   _metric |
|-------------------+------------+-------+----------+----------+--------+------------------+-----------|
| train_9ec57_00000 | TERMINATED |       | 0.44449  |        0 |      1 |      0.000305176 |  0.384359 |
| train_9ec57_00001 | TERMINATED |       | 0.44449  |        1 |      1 |      0.000396013 |  0.615227 |
| train_9ec57_00002 | TERMINATED |       | 0.44449  |        2 |      1 |      0.000357151 |  0.459991 |
| train_9ec57_00003 | TERMINATED |       | 0.44449  |        3 |      1 |      0.000265121 |  0.346182 |
| train_9ec57_00004 | TERMINATED |       | 0.774424 |        0 |      1 |      0.00022912  |  0.836283 |
| train_9ec57_00005 | TERMINATED |       | 0.774424 |        1 |      1 |      0.000292063 |  0.858715 |
| train_9ec57_00006 | TERMINATED |       | 0.774424 |        2 |      1 |      0.000271082 |  0.698797 |
| train_9ec57_00007 | TERMINATED |       | 0.774424 |        3 |      1 |      0.000377178 |  0.985665 |
+-------------------+------------+-------+----------+----------+--------+------------------+-----------+


2021-07-27 11:15:57,935	INFO tune.py:550 -- Total run time: 1.73 seconds (1.45 seconds for the tuning loop).
           _metric  time_this_iter_s  ...  iterations_since_restore  config/repeat
config/a                              ...                                         
0.444490  0.451439          0.000331  ...                       1.0            1.5
0.774424  0.844865          0.000292  ...                       1.0            1.5

[2 rows x 13 columns]
           _metric  time_this_iter_s  ...  iterations_since_restore  config/repeat
config/a                              ...                                         
0.444490  0.118994          0.000057  ...                       0.0       1.290994
0.774424  0.117510          0.000062  ...                       0.0       1.290994

[2 rows x 13 columns]

As you can see, the SD is around 0.11 (as expected, since we multiply by 0.1) and the mean is around the value of a (also as expected, since we add a to the standard normal sample value).

Does this help?