1st example given in keyconcepts docs doesn't appear to converge?

1

The first example of a search algorithm given on the introductory Key Concepts page doesn’t appear to work.

The code (comments and imports dropped for clarity) is:

def objective(x, a, b):
    return a * (x ** 0.5) + b

def trainable(config):
    for x in range(20):
        score = objective(x, config["a"], config["b"])
        tune.report(score=score)

config = {
    "a": tune.uniform(0, 1),
    "b": tune.uniform(0, 20)
}

hyperopt = HyperOptSearch(metric="score", mode="max")

tune.run(
    trainable,
    config=config,
    search_alg=hyperopt,
    num_samples=20,
    stop={"training_iteration": 20}
)

I would expect convergence to a=1, b=20, but actually it just looks like the search space is sampled at random for every trial.

  • One of the params of HyperOpt does say n_initial_points (int) – number of random evaluations of the objective function before starting to aproximate it with tree parzen estimators. Defaults to 20. which implies that the first 20 trials will be random, but increasing num_samples to 100 in tune.run still doesn’t seem to lead to convergence.

  • limiting concurrency with ConcurrencyLimiter(hyperopt, max_concurrent=1) doesn’t seem to make any difference

  • If I substitue the HyperOptSearch for BayesOptSearch AND limit concurrency then I do see convergence.

So I suppose my question is:

  • Is this a bug or am I missing something?
  • How can HyperOpt be used in a way that will intelligently converge upon optimal hyperparameters?

a sample run of hyperopt from the code above:

hyperopt
hyperopt1434×1503 349 KB

2

Hmm, thanks for typing up this writeup!

I just ran the example, and also pulled up the original hyperopt fmin usage: http://hyperopt.github.io/hyperopt/

By setting:

# Specify the search space and maximize score
hyperopt = HyperOptSearch(metric="score", mode="max")
hyperopt = ConcurrencyLimiter(hyperopt, max_concurrent=1)

# Makes the code run faster for simple functions
# ray.shutdown(); ray.init(local_mode=True)

# Execute 20 trials using HyperOpt and stop after 20 iterations
analysis = tune.run(
    trainable,
    config=config,
    search_alg=hyperopt,
    num_samples=100,
    verbose=1,
    stop={"training_iteration": 20})
    
print(objective(20, **analysis.get_best_config(metric="score", mode="max")))

I’m able to get similar results to Hyperopt’s fmin (which also doesn’t seem to converge exactly). So it seems like it’s not an issue with Ray but rather with Hyperopt. Maybe something like this thread may help for Hyperopt? https://github.com/hyperopt/hyperopt/issues/642

3

hmm ok. Can I suggest that the given example in the docs be something that works if copied and ran verbatim? It’s quite confusing otherwise.

Possibly this:

def objective(x, a, b):
    return a * (x ** 0.5) + b

def trainable(config):
    for x in range(20):
        score = objective(x, config["a"], config["b"])
        tune.report(score=score)

config = {
    "a": tune.uniform(0, 1),
    "b": tune.uniform(0, 20)
}

# use a simple naive bayes optimizer
opt = BayesOptSearch(metric="score", mode="max")

# by default, Ray will distribute trials across available computing resources
# naive bayes requires trial results sequentially
opt = ConcurrencyLimiter(opt, max_concurrent=1)

tune.run(
    trainable,
    config=config,
    search_alg=opt,
    num_samples=20,
    stop={"training_iteration": 10}
)
4

Yeah, that’s a good suggestion – thanks for bringing this up. Would you be interested in making an issue/PR?

Happy to help shepherd, just tag @richardliaw!