# Nested and conditional spaces TuneSearchCV

**URL:** <https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555>\
**Category:** Ray Tune\
**Created:** [June 17, 2021, 3:54pm UTC](https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555 "2021-06-17T15:54:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rmelvin](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rmelvin](https://discuss.ray.io/u/rmelvin)\
**Post date:** [June 17, 2021, 3:54pm UTC](https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555/1 "2021-06-17T15:54:02Z")

</div>

Hello! I want to run a conditional or nested search space through TuneSearchCV. Here’s an example of what I’d like to do

lda\_solver = tune.choice([‘svd’,‘lsqr’, ‘eigen’])  
lda\_shrinkiage = tune.sample\_from(lambda spec: None if spec.config.classify\_\_solver==‘svd’  
else (np.random.uniform(0,1)))

lda\_param\_grid ={  
“classify\_\_solver”: lda\_solver  
, “classify\_\_shrinkage”: lda\_shrinkiage  
}

tuned = TuneSearchCV(lda\_pipe,  
param\_distributions=lda\_param\_grid,  
n\_trials=300,  
scoring=“average\_precision”,  
max\_iters=1,  
search\_optimization=“bayesian”,  
n\_jobs=-1,  
refit=False,  
cv= 10,  
verbose=1,  
#loggers = “tensorboard”,  
random\_state=42,  
use\_gpu=False  
)  
tuned.fit(X\_train, y\_train)

But I get  
ValueError: SkOpt does not support parameters of type `Function` with samplers of type `NoneType`

Alternatively, it would also achieve what I want if I could set up a nested space like this:  
lda\_param\_grid = {  
‘solver’: tune.choice( [  
{‘solver’:‘svd’}  
,{‘solver’:‘lsqr’,‘shrinkage’:tune.choice([‘auto’, tune.uniform( 0, 1)])}  
,{‘solver’:‘eigen’,‘shrinkage’:tune.choice[‘auto’, tune.uniform( 0, 1)])}  
])  
}

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [June 17, 2021, 3:56pm UTC](https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555/2 "2021-06-17T15:56:20Z")

</div>

cc @Yard1 who is looking into improving conditional search spaces soon.

However, conditional search spaces are unlikely to work with custom searchers (such as BayesOpt, which you are using) anytime soon or ever. Can you try just passing `search_optimization="random"` instead?

---

<div class="post-metadata">

**Author:** ![rmelvin](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rmelvin](https://discuss.ray.io/u/rmelvin)\
**Post date:** [June 17, 2021, 4:04pm UTC](https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555/4 "2021-06-17T16:04:40Z")

</div>

I’m hoping for a “smarter” search method than random. I saw in the documentation other tune.suggest searchers can be passed for search\_optimization=…, but when I try something like  
from ray.tune.suggest.bayesopt import BayesOptSearch  
…  
search\_optimization=BayesOptSearch  
…

I get  
AttributeError: type object ‘BayesOptSearch’ has no attribute ‘lower’

---

<div class="post-metadata">

**Author:** ![Yard1](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yard1/32/508_2.png) [@Yard1](https://discuss.ray.io/u/Yard1)\
**Post date:** [June 17, 2021, 6:42pm UTC](https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555/5 "2021-06-17T18:42:12Z")

</div>

Hey @rmelvin , as Kai pointed out, we do not really support conditional parameters yet, and adding support for them will be tricky (and we won’t be able to add support for them to search algorithms that don’t support them internally). In the meantime, I was thinking about adding support for [Optuna define-by-run API](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html), which would allow you to specify your own function with conditions directly in the code. Would that be helpful in your usecase? Optuna supports bayesian optimization with TPEs.

---

<div class="post-metadata">

**Author:** ![rmelvin](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rmelvin](https://discuss.ray.io/u/rmelvin)\
**Post date:** [June 17, 2021, 8:03pm UTC](https://discuss.ray.io/t/nested-and-conditional-spaces-tunesearchcv/2555/6 "2021-06-17T20:03:00Z")

</div>

Hi @Yard1, that would help. My goal is to do a Bayesian search on a conditional search space. Thanks!