# Lightning- Early Stopping of training in Tune

**URL:** <https://discuss.ray.io/t/lightning-early-stopping-of-training-in-tune/8469>\
**Category:** Ray Tune stopping condition & comparisons\
**Created:** [November 29, 2022, 7:21pm UTC](https://discuss.ray.io/t/lightning-early-stopping-of-training-in-tune/8469 "2022-11-29T19:21:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mike1](https://avatars.discourse-cdn.com/v4/letter/m/e19adc/32.png) [@Mike1](https://discuss.ray.io/u/Mike1)\
**Post date:** [November 29, 2022, 7:21pm UTC](https://discuss.ray.io/t/lightning-early-stopping-of-training-in-tune/8469/1 "2022-11-29T19:21:15Z")

</div>

I have read [this guide](https://docs.ray.io/en/releases-1.11.0/tune/tutorials/tune-pytorch-lightning.html). In this guide, for each hyperparameter combination, it seems like Tune uses the metrics obtained by the network weights at the end of its training. However, I would like to use the network weights which yield the lowest validation score throughout training. For example, if the grid contains two hyperparameter combinations, and trains each of the two networks for 500 iterations, but the first network obtains the lowest validation score at iteration 70 and the second network obtains it at iteration 215, I want the grid search to compare the networks at their best points (iterations 70 and 215 respectively) instead of at iteration 500. For a single network, I know how to do that: use a ModelCheckpoint, and then use the best\_model\_path property. However, I don’t know how to make Tune do that. Can anyone help? Thank you!

---

<div class="post-metadata">

**Author:** ![Yard1](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yard1/32/508_2.png) [@Yard1](https://discuss.ray.io/u/Yard1)\
**Post date:** [November 29, 2022, 7:56pm UTC](https://discuss.ray.io/t/lightning-early-stopping-of-training-in-tune/8469/2 "2022-11-29T19:56:17Z")

</div>

Hey @Mike1, you can achieve that by configuring checkpointing in Tune to keep the best checkpoint per trial according to a metric. You can do that through the `keep_checkpoints_num` and `checkpoint_score_attr` arguments in `tune.run` API, or the `CheckpointConfig` object in the new, recommended `Tuner` API (available from Ray\>=2.0, you can see how to use it in the latest version of the documentation - [Using PyTorch Lightning with Tune — Ray 2.1.0](https://docs.ray.io/en/latest/tune/examples/tune-pytorch-lightning.html)).

Using the example I linked, you’d specify the `run_config` argument as:

```python
from ray.air.config import RunConfig, CheckpointConfig

        run_config=air.RunConfig(
            name="tune_mnist_asha",
            progress_reporter=reporter,
            checkpoint_config=CheckpointConfig(
                checkpoint_score_attribute="loss",
                checkpoint_score_order="min",
                # num_to_keep=1, # optionally set to only keep the best checkpoint on disk/cloud
            ),
        ),

```

Then, when you access the checkpoints after the run through the `checkpoint` attribute (eg. `results.get_best_result().checkpoint`), you will receive the checkpoint taken at the iteration which minimized the loss.

---

<div class="post-metadata">

**Author:** ![Mike1](https://avatars.discourse-cdn.com/v4/letter/m/e19adc/32.png) [@Mike1](https://discuss.ray.io/u/Mike1)\
**Post date:** [December 7, 2022, 7:14pm UTC](https://discuss.ray.io/t/lightning-early-stopping-of-training-in-tune/8469/3 "2022-12-07T19:14:04Z")

</div>

Thank you for the answer! However, I am not sure it does what I meant. It looks like results.get\_best\_result() still returns the network that got the best val loss at the end of training, not at the point where val loss was smallest, and the checkpoint returns the best val loss point for that network. For example: suppose I have two networks, net1 and net2, and:

loss(net1\_at\_end\_of\_training) \< loss(net2\_at\_end\_of\_training)  
loss(net1\_at\_best\_point\_during\_training) \> loss(net2\_at\_best\_point\_during\_training)

it seems that your code returns net1\_at\_best\_point\_during\_training, but I want something that returns net2\_at\_best\_point\_during\_training. Any suggestions?

---

<div class="post-metadata">

**Author:** ![Yard1](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yard1/32/508_2.png) [@Yard1](https://discuss.ray.io/u/Yard1)\
**Post date:** [December 7, 2022, 7:27pm UTC](https://discuss.ray.io/t/lightning-early-stopping-of-training-in-tune/8469/4 "2022-12-07T19:27:26Z")

</div>

Got it, thanks for clarifying! In that case, you want do to:  
`results.get_best_result(scope="all").checkpoint` - by default, `get_best_result` will only consider the last reported metric, but you can change the scope to consider all reports. Then, `checkpoint` will return the best checkpoint associated with the result.