Early termination?

1

Hi all

I’m evaluating Ray Tune for use for hyperparameter optimization and trying to get it to work with a toy problem - finding the maximum score on a simple parabola: score=b-(a-5)^{2}

The optimal hyperparameters are a=5, b=10 but after 11 Trials, Tune terminates early, having found a=6.3, b=10. why this early convergence on not a particularly good result when I explicitly ask for 20 trials?

The entire code and results are here:

from ray import tune
import hyperopt as hp
from ray.tune.suggest.hyperopt import HyperOptSearch
from ray.tune.suggest.bayesopt import BayesOptSearch

def objective(x, a, b):
    return b - ((a - 5) ** 2)

config = {
    "a": tune.uniform(0, 10),
    "b": tune.uniform(0, 10)
}

def trainable(config):
    for x in range(20):
        score = objective(x, config["a"], config["b"])
        tune.report(score=score)

bayesopt = BayesOptSearch(metric="score", mode="max")

analysis = tune.run(
    trainable,
    config=config,
    search_alg=bayesopt,
    num_samples=20,
    stop={"training_iteration": 20}
)

|Trial name |status |loc |a|b|iter|total time (s)|score|
|---|---|---|---|---|---|---|---|
|trainable_a17e3668|TERMINATED||3.7454 |9.50714|20|0.37529 |7.93312|
|trainable_a17e3669|TERMINATED||7.31994 |5.98658|20|0.356006|0.604466|
|trainable_a17e366a|TERMINATED||1.56019 |1.55995|20|0.350654|-10.2724|
|trainable_a17e366b|TERMINATED||0.580836|8.66176|20|0.19756 |-10.8672|
|trainable_a17e366c|TERMINATED||6.01115 |7.08073|20|0.304638|6.0583|
|trainable_a17e366d|TERMINATED||0.205845|9.6991 |20|0.27719 |-13.2848|
|trainable_a17e366e|TERMINATED||8.32443 |2.12339|20|0.262982|-8.92842|
|trainable_a17e366f|TERMINATED||1.81825 |1.83405|20|0.25714 |-8.28949|
|trainable_a17e3670|TERMINATED||3.04242 |5.24756|20|0.371475|1.41545|
|trainable_a17e3671|TERMINATED||4.31945 |2.91229|20|0.185346|2.44914|
|trainable_a17e373a|TERMINATED||6.31056 |10 |20|0.500303|8.28244|
2

This is probably due to parallelism causing the underlying model to improperly converge – can you try using a ConcurrencyLimiter?

3

Hi, thanks for reply! A ConcurrencyLimiter with max_concurrent=1 does indeed stop early termination, settling on a more reasonable a=5.063, b=10 after 20 trials.

I also note that ConcurrencyLimiter says

Use ray.tune.suggest.ConcurrencyLimiter to limit the amount of concurrency when using a search algorithm. This is useful when a given optimization algorithm does not parallelize very well (like a naive Bayesian Optimization).

This behaviour has thrown me a little though - is this a bug or intended/expected behaviour? If this is intended, would you mind explaining a little? I can loosely understand why iteratively converging to a maxima with a bayesian methodology might not parallelize in a particularly efficient way, since the results of many trials that come back will be from slightly outdated hypotheses. but it’s still fundamentally novel information that the optimizer ought to be able to incorporate - why would it terminate early instead?