Hot to use together ray tune+ xgboost+cross_val_score+early stopping

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hi,
I try to use together ray tune+ xgboost+cross_val_score+early stopping and I have the problem how to add early stopping.

I can use:
ray tune+ xgboost+cross_val_score or ray tune+ xgboost +early stopping
but I don’t know how tu use both cross_val_score and early stopping

I found some tutorials:
https://docs.ray.io/en/master/tune/tutorials/tune-xgboost.html#early-stopping

I try to use together ray tune+ xgboost+cross_val_score+early stopping and I have the problem how to add early stopping.

I can use:
ray tune+ xgboost+cross_val_score or ray tune+ xgboost +early stopping
but I don’t know how tu use both cross_val_score and early stopping in the same ray tuning

I found tutorials:
https://docs.ray.io/en/master/tune/tutorials/tune-xgboost.html#early-stopping

https://stackoverflow.com/questions/43866284/grid-search-and-early-stopping-using-cross-validation-with-xgboost-in-scikit-lea

10. XGBoost with Hyperopt, Optuna, and Ray
and code:

fit_params = {"early_stopping_rounds":1,
             "eval_set" : [[testX, testY]]}


model = xgb.XGBRegressor()
gridsearch = GridSearchCV(estimator  = xgb.XGBRegressor(), 
                          param_grid=param_grid,
                          fit_params=fit_params,                          
                          verbose=1,                          
                          cv=2)

I’m not sure how to use “eval_set” if I make cross validation . I will be grateful for any hints or ideas.

2

Hey @Peter_Pirog, have you considered using tune-sklearn? It has automatic support for XGBoost early stopping. You can install it through pip install tune-sklearn, and use it in a similar way to the example here: tune-sklearn/xgbclassifier.py at master · ray-project/tune-sklearn · GitHub

It will automatically run both cross validation and early stopping, up to the value of the n_estimators parameter of the passed XGBRegressor, and you don’t need to pass any fit_params.

3

@Yard1 Thank You for the answer. I will try it. :grinning:
I will check paremetwers description in https://docs.ray.io/en/master/tune/api_docs/sklearn.html?highlight=tune-sklearn#

1 Like