Best tuning algorithm for dealing with real and integer values

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

When dealing with the tuning of both real and integer values, what optimizer technique should be used?

2

Real and categorical values tend to be well supported by HyperOpt and Ax, if I remember correctly.

3

Thanks @rliaw for your answer!
I have tried to dig deeper for each optimizer to make the question more specific.
Generally speaking, it seems to me that the tune.choice modelling options fail to capture the order of a discrete integer. Is there for instance something like tune.discrete_uniform?

HyperOpt

The HyperOpt optimizer supports integer ranges in the space by using the following snippet:

hp.randint(label, upper)

In your opinion, does this option capture the order of the distribution?

Ax

For what I am seeing in the documentation of Ax, both in the provided example and in the documentation example I only see examples of continuous space. Does it actually support also tune.choice? Is there a specific way to model discrete integer distributions? I have found that in their code the following example:

RangeParameter("a", lower=1, upper=2, parameter_type=ParameterType.INT),

Other optimizers that support the choice method

I see that there are other optimizers that support the choice method, such as:

BOHB

The BOHB optimizer supports the choice parameter, but I seem to remember that it was preferable to use instead the BayesOptSearch alongside with ASHA. Does this still stand?

Nevergrad

The Nevergrad optimizer supports the choice parameter as well. How does this optimizer compare with HyperOpt? Is there some different scaling, either in terms of parallel jobs or number of hyper-parameters?
Also, in this case, I could find in their documentation an example on how to use integer values here.

ng.p.Scalar(lower=1, upper=12).set_integer_casting()

ZOOpt

The ZOOpt optimizer seems to support actually integer distributions by using the following hyper-parameters space:

{
    "width": (ValueType.DISCRETE, [-10, 10], False),
}

It is model by tune using instead the tune.uniform(-10, 10), which would seem to be to lose the discrete aspect of the aforementioned distribution. Am I mistaken?

DragonFly

The DragonFly optimizer does not seem to support integer parameters from the Ray documentation, but it seems to be possible according to the example they have provided in their GitHub repository. Is this supported in the Ray wrapper?

{
"n_estimators" : {
    "name":"n_estimators",
    "type":"int",
    "min":1,
    "max":1000
}
}
4

I did not yet applied the BOHB in my practise, but I look forward to using it after having read this blogpost by the authors BOHB: Robust and Efficient Hyperparameter Optimization at Scale.

5

Ah, I think Tune has a tune.randint call that you might be looking for!

6

Thank you, I will try this! How does tune.randint behave when dealing with libraries such as BayesianOptimization that, to my knowledge, do not support discrete sampling spaces?

7

I think it might just throw an error for those situations (cc @kai)

8

Yes, Ray Tune will let you know when a search space cannot be converted to the native search space of any search algorithm.

9

Do you think it would be a nice feature to allow for automatic “downcasting”? Like, for optimizers that do not support randint switch to choice and, if necessary, switch to uniform?

10

Hmm, it would be an interesting proposal but it is a little tricky to implement (and definitely not to be enabled by default)

11

Sure, it would simply allow for a bit more ‘easy to use’ code. Possibly it could be a parameter of the tune.method(), something like tune.method(..., auto_downcast=True).