# Tuning xgboost with early stopping

**URL:** <https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75>\
**Category:** Ray Tune\
**Created:** [November 26, 2020, 7:42am UTC](https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75 "2020-11-26T07:42:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![royalts](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/royalts/32/44_2.png) [@royalts](https://discuss.ray.io/u/royalts)\
**Post date:** [November 26, 2020, 7:42am UTC](https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75/1 "2020-11-26T07:42:45Z")

</div>

I’m trying to perform hyperparameter tuning for an xgboost classifier. I gather there are at least two ways to do this and I’m trying to figure out what exactly they do and how they might differ:

(1) tune.run() with the native xgboost interface ([https://docs.ray.io/en/latest/tune/tutorials/tune-xgboost.html](https://docs.ray.io/en/latest/tune/tutorials/tune-xgboost.html))  
(2) TuneSearchCV using the sklearn interface to xgboost ([https://github.com/ray-project/tune-sklearn/blob/master/examples/xgbclassifier.py](https://github.com/ray-project/tune-sklearn/blob/master/examples/xgbclassifier.py)).

How exactly does early stopping work in either of those cases? Are there differences between these two implementations?

If, say, I run (2) with a classifier with n\_estimator=500 and set early\_stopping=True and n\_trials=20 then tune will run 20 parameter combinations but not for the full 500 boosting rounds, correct?

But what determines after how many rounds the score will be checked and which trials will be prematurely abandoned? Is there a way for me to see after the fact how each of these trials was handled, i.e. when it was killed and why?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [November 26, 2020, 8:05am UTC](https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75/2 "2020-11-26T08:05:13Z")

</div>

Hey @royalts thanks for dropping by!

1. In tune.run case, early stopping will occur between boosting rounds (assuming you use the xgboost callback in Tune). The early stopping decision will be made by with any [Tune scheduler](https://docs.ray.io/en/master/tune/api_docs/schedulers.html) you choose.
2. in tune-sklearn, we actually [implement incremental fitting](https://github.com/dmlc/xgboost/issues/1686#issuecomment-357187570) for xgboost models. The decision for stopping by default is with [ASHA](https://docs.ray.io/en/master/tune/api_docs/schedulers.html#tune-scheduler-hyperband), but you can provide any arbitrary Tune scheduler. You can set `TuneSearchCV(verbose=...` to see how/when decisions are made.

Does that help?

---

<div class="post-metadata">

**Author:** ![royalts](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/royalts/32/44_2.png) [@royalts](https://discuss.ray.io/u/royalts)\
**Post date:** [November 26, 2020, 8:29am UTC](https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75/3 "2020-11-26T08:29:36Z")

</div>

That helped a lot! Mostly consolidated the understanding I had cobbled together from various docs. I hope I can ask some more clarifying questions:

- So in (2) each trial will run xgboost for m boosting rounds, then check the score (which score on which dataset? mean test?) and then make a termination decision. If never terminated the trial will run until `n_estimator` rounds? How do I set m?
- Is there a way to get at the trial histories through things other than `verbose` logs?

---

<div class="post-metadata">

**Author:** ![royalts](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/royalts/32/44_2.png) [@royalts](https://discuss.ray.io/u/royalts)\
**Post date:** [November 26, 2020, 5:03pm UTC](https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75/4 "2020-11-26T17:03:35Z")

</div>

Hmm, perhaps that lingo was a bit too xgboost-specific. What I’m interested in is how to set the batch size for each of the incremental fits and how to set the maximal total number of rounds.

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [November 28, 2020, 6:14pm UTC](https://discuss.ray.io/t/tuning-xgboost-with-early-stopping/75/5 "2020-11-28T18:14:45Z")

</div>

Hmm,

- To set “m”, you can set parameters for your [scheduler](https://docs.ray.io/en/master/tune/api_docs/schedulers.html#asha-tune-schedulers-ashascheduler)
- If never terminated, yes I think the trial will run until `n_estimator` rounds.
- To get trial history, you can set a logger (see the TuneSearchCV(loggers=…)) parameter.

Hopefully that helps!