How to pick up hyper params random on each iter

[2w.zend]

Hi,
I’m a bit confused,
how to get random/suggested values of p1, p2 during each iteration?

import ray
from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.schedulers import AsyncHyperBandScheduler
from ray.tune.suggest.optuna import OptunaSearch

def easy_objective(config):
    for step in range(config["iters"]):
        p1, p2 = config["p1"], config["p2"]
        print('P1', p1, 'P2', p2)
        intermediate_score = p1 + p2

        tune.report(iterations=step, score=intermediate_score)


if __name__ == "__main__":
    ray.init(configure_logging=False)
    algo = OptunaSearch()
    algo = ConcurrencyLimiter(algo, max_concurrent=1)
    scheduler = AsyncHyperBandScheduler()
    analysis = tune.run(
        easy_objective,
        num_samples=1,
        metric="score",
        mode="max",
        search_alg=algo,
        scheduler=scheduler,
        config={
            "iters": 10,
            "p1": tune.randint(0, 10),
            "p2": tune.randint(0, 10),
        })

    print("Best hyperparameters found were: ", analysis.best_config)

[kai]

Hyperparameters are usually fixed at the beginning of each trial. That means that config["p1"] and config["p2"] won’t change.

Only in some algorithms (like Population Based Training) are hyperparameters changed during training.

What are you trying to do/optimize exactly?

[2w.zend]

I have evaluation function, which return score based on p1,p2,p3,p4,p5.
So i need to find out best value for params (all int).
Of course eval_func more complicated than just p1+p2+p3+p4+p5.
I’m a bit lost in all configs, any suggestion how to implement this ?

def eval_func(p1, p2, p3, p4, p5):
    return p1+p2+p3+p4+p5

[Peter_Pirog]

@2w.zend

import ray
import numpy as np
from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.schedulers import AsyncHyperBandScheduler
from ray.tune.suggest.optuna import OptunaSearch

def easy_objective(config):
    for step in range(config["iters"]):
        p1=np.random.randint(config["p1"])
        p2=np.random.randint(config["p2"])

        print('P1', p1, 'P2', p2)
        intermediate_score = p1 + p2

        tune.report(iterations=step, score=intermediate_score)


if __name__ == "__main__":
    ray.init(configure_logging=False)
    algo = OptunaSearch()
    algo = ConcurrencyLimiter(algo, max_concurrent=1)
    scheduler = AsyncHyperBandScheduler()
    analysis = tune.run(
        easy_objective,
        num_samples=1,
        metric="score",
        mode="max",
        search_alg=algo,
        scheduler=scheduler,
        config={
            "iters": 10,
            "p1": tune.choice([10]),
            "p2": tune.choice([10]),
        })

My suggestion how you can do that.

[kai]

I see, so you’re essentially trying to evaluate a function at 10 different points?

If so, then you would usually start 10 different trials (num_samples argument in tune.run()) and have them each evaluate one point, like so:

import ray
from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.schedulers import AsyncHyperBandScheduler
from ray.tune.suggest.optuna import OptunaSearch

def easy_objective(config):
    p1, p2 = config["p1"], config["p2"]
    print('P1', p1, 'P2', p2)
    intermediate_score = p1 + p2
    tune.report(iterations=step, score=intermediate_score)


if __name__ == "__main__":
    ray.init(configure_logging=False)
    algo = OptunaSearch()
    algo = ConcurrencyLimiter(algo, max_concurrent=1)
    scheduler = AsyncHyperBandScheduler()
    analysis = tune.run(
        easy_objective,
        num_samples=10,
        metric="score",
        mode="max",
        search_alg=algo,
        scheduler=scheduler,
        config={
            "p1": tune.randint(0, 10),
            "p2": tune.randint(0, 10),
        })

    print("Best hyperparameters found were: ", analysis.best_config)

[2w.zend]

Hi,

1. Is there any other option?
   If I use num_samples it’s slower at least x1000 times then using Optuna directly.
   For example, 1000 trials with optuna complete in 47 seconds.
   But using ray it’s took > 1 hour.
2. Am I right in this case every time ray will pick random points. Is there a way ray will try use best values accordingly to previous trials ? Or such magic doesn’t exists ?

[kai]

Regarding 1, we did a couple of improvements in recent weeks to speed this up. Which version of Ray are you currently using? It seems your evaluation function takes less than a second to complete, is that correct? I’m just asking because we specifically want to make sure large scale experiments work smoothly, and if it doesn’t work for you we’re happy to look into this further.

Regarding 2, point selection depends on the search algorithm you use. In the example you are using OptunaSearch, which uses Tree-Parzen estimators per default. This sampler should pick points according to results it observed in previous trials. In fact, all search algorithms except for grid search and random search (default if you pass no search_alg argument) should adapt to previously seen evaluations.

[2w.zend]

1. I’m using ray, version 1.2.0.dev0 (from git). Eval function take roughly 0.005 ms.
   If I can help anyhow, I’m gladly will send debug info.

2. Here is a simple example, best hyperparameters should be ‘p1’: 100, ‘p2’: 100. But 9 out of 10 times I run it, best values found by ray it’s way away of being ‘p1’: 100, ‘p2’: 100. What am I doing wrong?

import ray
from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.suggest.optuna import OptunaSearch

def easy_objective(config):
    p1, p2 = config["p1"], config["p2"]
    print('P1', p1, 'P2', p2)
    return {'score': p1+p2}


if __name__ == "__main__":
    ray.init(configure_logging=False)
    algo = OptunaSearch()
    algo = ConcurrencyLimiter(algo, max_concurrent=1)
    analysis = tune.run(
        easy_objective,
        num_samples=20,
        metric="score",
        mode="max",
        search_alg=algo,
        config={
            "iters": 10,
            "p1": tune.randint(0, 100),
            "p2": tune.randint(0, 100),
        })

    print("Best hyperparameters found were: ", analysis.best_config)

[kai]

Ah I see.

So a good way to speed up your training is to enable reuse_actors=True in tune.run():

import time
import ray
from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.suggest.optuna import OptunaSearch

def easy_objective(config):
    p1, p2 = config["p1"], config["p2"]
    print('P1', p1, 'P2', p2)
    return {'score': p1+p2}


if __name__ == "__main__":
    ray.init(configure_logging=False)
    algo = OptunaSearch()
    algo = ConcurrencyLimiter(algo, max_concurrent=1)
    start_time = time.time()
    analysis = tune.run(
        easy_objective,
        num_samples=100,
        metric="score",
        mode="max",
        search_alg=algo,
        config={
            "iters": 10,
            "p1": tune.randint(0, 100),
            "p2": tune.randint(0, 100),
        },
        reuse_actors=True)
    time_taken = time.time() - start_time

    print(f"Took {time_taken:.2f} seconds.")

    print("Best hyperparameters found were: ", analysis.best_config)

When training 100 trials
Without reuse_actors: Took 32.28 seconds.
With reuse_actors: Took 2.44 seconds.

On my MacBook, 1000 trials with this trainable take 27.07 seconds.

Regarding your second question, 100 trials finds the best solution easily:

Best hyperparameters found were:  {'iters': 10, 'p1': 100, 'p2': 100}

With 10 trials this is sometimes a bit off (e.g. {'iters': 10, 'p1': 89, 'p2': 78}). Most search algorithms try random sampling at first and then converge to a more accurate representation of the objective function. I’d have to look closer into Optuna’s implementation. Generally you shouldn’t get results that are too far off after a number of trials (say, 20 or 30 or so).