How to use max_iters with TuneSearchCV, XGBRegressor, and BOHB

RLlib
1

I’m not sure I understand how the max_iters parameter is being implemented with TuneSearchCV, an XGBRegressor model, and bohb as the optimizer.

Per the TuneSearchCV documentation,

Depending on the classifier type provided, a resource parameter (resource_param = max_iter or n_estimators) will be detected. The value of resource_param will be treated as a “max resource value”, and all classifiers will be initialized with max resource value // max_iters, where max_iters is this defined parameter. On each epoch, resource_param (max_iter or n_estimators) is incremented by max resource value // max_iters.

So, if one of the parameters that I’m optimizing for is n_iterations and I don’t set a max_iters value, then I would expect each epoch to use n_estimators iterations. However, in practice, I can’t really tell if that’s the case based on the console output.

With search_n_trials=200, 200 rounds are performed, each with the following bracket info (#159 is just an example):

Round #159:
  Bracket(Max Size (n)=1, Milestone (r)=1, completed=100.0%): {TERMINATED: 1}

Is this expected behavior? Should I be manually setting max_iters even though I’m also optimizing for n_estimators with the XGBoost model? I would have expected n to be different, but admittedly I don’t fully understand Ray’s implementation of bohb within tune-sklearn.

Here’s my implementation:

param_space = {
        "n_estimators": tune.lograndint(100, 2000),
        "max_depth": tune.randint(2, 6),
        "subsample": tune.quniform(0.25, 0.75, 0.01),
        "colsample_bytree": tune.quniform(0.05, 0.5, 0.01),
        "colsample_bylevel": tune.quniform(0.05, 0.5, 0.01),
        "learning_rate": tune.quniform(1e-3, 1e-1, 5e-4),
    }

    xgb_model = XGBRegressor(n_jobs=1, booster="gbtree")

    reg = TuneSearchCV(
        xgb_model,
        param_space,
        n_trials=200,
        scoring={"rmse": "neg_root_mean_squared_error", "r2": "r2"},
        refit="rmse",
        n_jobs=-1,
        cv=KFold(5),
        verbose=1,
        random_state=42,
        search_optimization="bohb"
    )

    reg.fit(X, y)