# Best tuning algorithm for dealing with real and integer values

**URL:** <https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455>\
**Category:** Ray Tune\
**Created:** [January 15, 2021, 5:25pm UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455 "2021-01-15T17:25:40Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![LucaCappelletti94](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucacappelletti94/32/144_2.png) [@LucaCappelletti94](https://discuss.ray.io/u/LucaCappelletti94)\
**Post date:** [January 15, 2021, 5:25pm UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/1 "2021-01-15T17:25:40Z")

</div>

When dealing with the tuning of both real and integer values, what optimizer technique should be used?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 15, 2021, 6:54pm UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/2 "2021-01-15T18:54:15Z")

</div>

Real and categorical values tend to be well supported by HyperOpt and Ax, if I remember correctly.

---

<div class="post-metadata">

**Author:** ![LucaCappelletti94](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucacappelletti94/32/144_2.png) [@LucaCappelletti94](https://discuss.ray.io/u/LucaCappelletti94)\
**Post date:** [January 16, 2021, 9:20am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/3 "2021-01-16T09:20:35Z")

</div>

Thanks @rliaw for your answer!  
I have tried to dig deeper for each optimizer to make the question more specific.  
Generally speaking, it seems to me that the `tune.choice` modelling options fail to capture the order of a discrete integer. Is there for instance something like `tune.discrete_uniform`?

### HyperOpt

The [HyperOpt optimizer](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#hyperopt-tune-suggest-hyperopt-hyperoptsearch) supports [integer ranges in the space](https://github.com/hyperopt/hyperopt/blob/3372a16e9f9a39f56f072dc6f90510bcd8e3758f/docs/templates/getting-started/search_spaces.md) by using the following snippet:

```python
hp.randint(label, upper)

```

In your opinion, does this option capture the order of the distribution?

### Ax

For what I am seeing in the documentation of Ax, [both in the provided example](https://docs.ray.io/en/master/tune/examples/ax_example.html) and in the [documentation example](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#tune-ax) I only see examples of continuous space. Does it actually support also `tune.choice`? Is there a specific way to model discrete integer distributions? I have found that [in their code](https://github.com/facebook/Ax/blob/02b435132ea5f836ab0a7408422e0cb5855948cf/ax/modelbridge/tests/test_ordered_choice_encode_transform.py#L24) the following example:

```python
RangeParameter("a", lower=1, upper=2, parameter_type=ParameterType.INT),

```

### Other optimizers that support the choice method

I see that there are other optimizers that support the choice method, such as:

#### BOHB

The [BOHB](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#bohb-tune-suggest-bohb-tunebohb) optimizer supports the choice parameter, but I seem to remember that it was preferable to use instead the [BayesOptSearch](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#bayesopt) alongside with [ASHA](https://docs.ray.io/en/master/tune/api_docs/schedulers.html?highlight=ASHA#asha-tune-schedulers-ashascheduler). Does this still stand?

#### Nevergrad

The [Nevergrad optimizer](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#nevergrad-tune-suggest-nevergrad-nevergradsearch) supports the choice parameter as well. How does this optimizer compare with HyperOpt? Is there some different scaling, either in terms of parallel jobs or number of hyper-parameters?  
Also, in this case, I could find in their documentation an example [on how to use integer values here](https://facebookresearch.github.io/nevergrad/index.html?highlight=integer).

```python
ng.p.Scalar(lower=1, upper=12).set_integer_casting()

```

#### ZOOpt

The [ZOOpt optimizer](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#zoopt-tune-suggest-zoopt-zooptsearch) seems to support actually integer distributions by using the following hyper-parameters space:

```python
{
    "width": (ValueType.DISCRETE, [-10, 10], False),
}

```

It is model by tune using instead the `tune.uniform(-10, 10)`, which would seem to be to lose the discrete aspect of the aforementioned distribution. Am I mistaken?

#### DragonFly

The [DragonFly optimizer](https://docs.ray.io/en/master/tune/api_docs/suggestion.html?highlight=BayesOptSearch#dragonfly-tune-suggest-dragonfly-dragonflysearch) does not seem to support integer parameters from the Ray documentation, but it seems to be possible according to [the example they have provided in their GitHub repository](https://github.com/dragonfly/dragonfly/blob/a579b5eadf452e23b07d4caf27b402703b0012b7/examples/synthetic/borehole_6/config_mf.json#L25). Is this supported in the Ray wrapper?

```python
{
"n_estimators" : {
    "name":"n_estimators",
    "type":"int",
    "min":1,
    "max":1000
}
}

```

---

<div class="post-metadata">

**Author:** ![gregoruar](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gregoruar/32/224_2.png) [@gregoruar](https://discuss.ray.io/u/gregoruar)\
**Post date:** [January 19, 2021, 4:00pm UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/4 "2021-01-19T16:00:41Z")

</div>

I did not yet applied the BOHB in my practise, but I look forward to using it after having read this blogpost by the authors [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://www.automl.org/blog_bohb/).

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 25, 2021, 8:10am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/5 "2021-01-25T08:10:41Z")

</div>

Ah, I think Tune has a `tune.randint` call that you might be looking for!

---

<div class="post-metadata">

**Author:** ![LucaCappelletti94](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucacappelletti94/32/144_2.png) [@LucaCappelletti94](https://discuss.ray.io/u/LucaCappelletti94)\
**Post date:** [January 25, 2021, 8:36am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/6 "2021-01-25T08:36:30Z")

</div>

Thank you, I will try this! How does tune.randint behave when dealing with libraries such as BayesianOptimization that, to my knowledge, do not support discrete sampling spaces?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 25, 2021, 8:40am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/7 "2021-01-25T08:40:15Z")

</div>

I think it might just throw an error for those situations (cc @kai)

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [January 25, 2021, 8:57am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/8 "2021-01-25T08:57:43Z")

</div>

Yes, Ray Tune will let you know when a search space cannot be converted to the native search space of any search algorithm.

---

<div class="post-metadata">

**Author:** ![LucaCappelletti94](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucacappelletti94/32/144_2.png) [@LucaCappelletti94](https://discuss.ray.io/u/LucaCappelletti94)\
**Post date:** [January 25, 2021, 9:12am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/9 "2021-01-25T09:12:02Z")

</div>

Do you think it would be a nice feature to allow for automatic “downcasting”? Like, for optimizers that do not support randint switch to choice and, if necessary, switch to uniform?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 26, 2021, 9:12am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/10 "2021-01-26T09:12:20Z")

</div>

Hmm, it would be an interesting proposal but it is a little tricky to implement (and definitely not to be enabled by default)

---

<div class="post-metadata">

**Author:** ![LucaCappelletti94](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucacappelletti94/32/144_2.png) [@LucaCappelletti94](https://discuss.ray.io/u/LucaCappelletti94)\
**Post date:** [January 30, 2021, 10:09am UTC](https://discuss.ray.io/t/best-tuning-algorithm-for-dealing-with-real-and-integer-values/455/11 "2021-01-30T10:09:31Z")

</div>

Sure, it would simply allow for a bit more ‘easy to use’ code. Possibly it could be a parameter of the `tune.method()`, something like `tune.method(..., auto_downcast=True)`.