# Understanding stop training\_iteration parameter

**URL:** <https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597>\
**Category:** Ray Tune\
**Created:** [September 22, 2021, 3:38pm UTC](https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597 "2021-09-22T15:38:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vblagoje/32/1442_2.png) [@vblagoje](https://discuss.ray.io/u/vblagoje)\
**Post date:** [September 22, 2021, 3:38pm UTC](https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597/1 "2021-09-22T15:38:18Z")

</div>

Hi,

I am trying to understand training\_iteration in the context of stopping the trial run. If I don’t specify the stopping condition, the trials run for 100 iterations. If I specify a value lower than 100, say:

```auto
analysis = tune.run(
        trainable_cls,
        scheduler=scheduler,
        num_samples=100,
        search_alg=algo,
        config=config,
        stop={"training_iteration": 32},
        metric="loss",
        mode="min")

```

The value is respected and the trial ends at 32 iterations. However, if I change this condition to a value above 100 the trials still stop at 100. How can I raise the number of iterations?

Thanks,  
Vladimir

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [September 22, 2021, 6:07pm UTC](https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597/2 "2021-09-22T18:07:29Z")

</div>

Hey @vblagoje, can you confirm from the output (example below) that what you’re seeing is that each of the 100 (`num_samples`) trials is stopping at `iter` 100 when `training_iteration` is set higher?

```auto
+-----------------------+------------+-------+--------+------------------+
| Trial name | status | loc | iter | total time (s) |
|-----------------------+------------+-------+--------+------------------|
| Trainable_5678d_00000 | TERMINATED | | 200 | 0.000630856 |
+-----------------------+------------+-------+--------+------------------+

```

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vblagoje/32/1442_2.png) [@vblagoje](https://discuss.ray.io/u/vblagoje)\
**Post date:** [September 23, 2021, 6:51am UTC](https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597/3 "2021-09-23T06:51:21Z")

</div>

Hey @matthewdeng ,

Not every trial run stops at 100 as I am running Bayesian search with the AsyncHyperBandScheduler scheduler. A few of them stopped at 100, many at 1 and a few around 10-15. Here is my full config:

```auto
    algo = BayesOptSearch(metric="loss", mode="min")
    algo = ConcurrencyLimiter(algo, max_concurrent=1)
    scheduler = AsyncHyperBandScheduler()
    trainable_cls = DistributedTrainableCreator(
        distributed_bert_pretraining,
        num_workers=config["n_gpu"],
        backend="nccl",
        num_gpus_per_worker=1)

    tune_config = {
        "phase1_learning_rate": tune.uniform(1e-5, 1e-3),
        "weight_decay": tune.uniform(1e-4, 1e-1),
        "warmup_proportion": tune.uniform(0.01, 0.20)
        }
    config = { **config,** tune_config}
    print(f"Config for the tune is {config}")
    analysis = tune.run(
        trainable_cls,
        scheduler=scheduler,
        num_samples=50,
        search_alg=algo,
        config=config,
        stop={"training_iteration": 320},
        metric="loss",
        mode="min")

```

Thanks in advance,  
Vladimir

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [September 23, 2021, 5:37pm UTC](https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597/4 "2021-09-23T17:37:47Z")

</div>

Thanks for sharing the code! This is because of the default values of [AsyncHyperBandScheduler](https://docs.ray.io/en/latest/_modules/ray/tune/schedulers/async_hyperband.html#AsyncHyperBandScheduler):

```python
                 time_attr: str = "training_iteration",
                 max_t: int = 100,

```

You can increase the number of iterations by passing in a larger `max_t` when initializing the `AsyncHyperBandScheduler`.

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vblagoje/32/1442_2.png) [@vblagoje](https://discuss.ray.io/u/vblagoje)\
**Post date:** [September 24, 2021, 7:49am UTC](https://discuss.ray.io/t/understanding-stop-training-iteration-parameter/3597/5 "2021-09-24T07:49:24Z")

</div>

Hey @matthewdeng,  
You are right! Thanks again and all the best.