Search space config Validation for Search algorithms

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

I was wondering if we could add validation of search space config for the search algorithms.
For example, grid_search is not supported for AxSearch , BayesOpt doesn’t support Integer data type. It raises an error during the tune.run .
If there could be a way to get functionalities that are not supported for search_alg before running tune.run. I believe it would be helpful.
Thank you!

2

Yeah, I agree this would really improve usability. cc @kai

@kaushikb11 would you be interested in making a contribution?

3

Yeah, sure! Do you have anything in mind for the implementation? Will go through the code.

4

I think every class has a set_search_space option. Maybe there you can raise an error if a grid search is passed in without support?

5

A question I have here is how you would want to get this information.

Usually the config parameter is passed to tune.run (e.g. tune.run(config=config)). Do I understand that you would like to have something like AxSearch.is_valid_search_space(config) here?

If so I would suggest to add a method to the base Searcher class that tries to convert the search space and errors if it is invalid. Something like this:

@staticmethod
def is_valid_search_space(spec: Dict):
    try:
        this.convert_search_space(spec)
    except ValueError:
        return False
    return True

This would keep it really simple and should avoid code duplication.

6

@kai This makes sense. Thanks! Will look into it.

I faced an error in convert_search_space for AxSearch search alg: Issue