# Question - About tune stopping condition with PBT

**URL:** <https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447>\
**Category:** Uncategorized\
**Created:** [February 21, 2023, 5:47pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447 "2023-02-21T17:47:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alana](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alana/32/3933_2.png) [@Alana](https://discuss.ray.io/u/Alana)\
**Post date:** [February 21, 2023, 5:47pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/1 "2023-02-21T17:47:05Z")

</div>

Hi there !  
I hope you’re having a great day.

I have some questions about population based training with ray, and more specifically the moment it stops to tune.

For more context, here’s a part of my code (I’m using tune.run which I know is the soon-to-be deprecated API):

```auto
         scheduler = PopulationBasedTraining(
             hyperparam_mutations=hyperparam_mutations,
             time_attr="training_iteration",
             metric="val_loss",
             mode="min",
             perturbation_interval=1,
         )
 
         reporter = CLIReporter(
             parameter_columns=parameters_to_display,
             metric_columns=["val_loss", "epoch"],
             metric="val_loss",
             mode="min"
         )
 
         results = tune.run(
             partial(training_func),
             config=config,
             scheduler=scheduler,
             num_samples=2,
             progress_reporter=reporter,
             checkpoint_score_attr="training_iteration",
             keep_checkpoints_num=1,
             name=experience,
             checkpoint_at_end=True,
             local_dir="./ray_results",
             log_to_file=True, 
             resources_per_trial={"cpu": args.cpu, "gpu": args.gpu},
             resume="AUTO", 
             sync_config=tune.SyncConfig(syncer=None),
         )

```

Here, syncer is set to None as I’m using this code within SLURM cluster manager.

I’m using a function API (training\_func) where I train over X number of epochs (and not an infinity loop). I use session.report and checkpointing at the end of each epoch (for registering info inside tensorboardX).

From my understanding, it seems that because I only train over X epochs, a trial is considered done when it reach the last epoch and it exit the training\_func. When all trials (here 2) are finished, the run is finished and it stops tuning.  
Is that correct ?

If so, is there any way to continue tuning (by maybe re-setting trials periodically) and only stop it after some conditions are met (with stop option) ? Should I use more trials (increase num\_samples) ? If the solution is to use an infinity loop instead inside my training\_func (with something like while True), is there a way to prevent trials to reach past a certain epoch (I need to compare results at a defined epoch) ? Or to retrieve best trial for a precise epoch ?

Thanks for your insight !

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [February 21, 2023, 6:48pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/2 "2023-02-21T18:48:35Z")

</div>

Yes, your understanding is correct. Could you try setting X to higher number?

---

<div class="post-metadata">

**Author:** ![Alana](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alana/32/3933_2.png) [@Alana](https://discuss.ray.io/u/Alana)\
**Post date:** [February 21, 2023, 7:01pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/3 "2023-02-21T19:01:48Z")

</div>

Thanks @xwjiang2010 for your answer !

I can’t really modify the number of training epochs for a comparison purpose (I want to compare one model to another after the same number of epochs).  
I’ll try to increase my number of trials to get better results.  
Or is there a way to get the best trial for a defined epoch/training\_iteration ?

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [February 21, 2023, 7:49pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/4 "2023-02-21T19:49:53Z")

</div>

Can we decouple tuning iterations and getting relevant results? For tuning, you can set a big X. And then can you use this [API](https://docs.ray.io/en/latest/tune/api_docs/result_grid.html#result-air-result) to get metrics for all trials and all iterations. And then you can do all comparisons you want.

---

<div class="post-metadata">

**Author:** ![Alana](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alana/32/3933_2.png) [@Alana](https://discuss.ray.io/u/Alana)\
**Post date:** [February 21, 2023, 8:05pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/5 "2023-02-21T20:05:27Z")

</div>

Thanks again for you answer and the API link =)

---

<div class="post-metadata">

**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\
**Post date:** [February 21, 2023, 8:43pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/6 "2023-02-21T20:43:55Z")

</div>

@Alana This guide may also be useful to you! [Analyzing Tune Experiment Results — Ray 3.0.0.dev0](https://docs.ray.io/en/master/tune/examples/tune_analyze_results.html#trial-level-analysis-working-with-an-individual-result)

---

<div class="post-metadata">

**Author:** ![Alana](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alana/32/3933_2.png) [@Alana](https://discuss.ray.io/u/Alana)\
**Post date:** [February 21, 2023, 9:25pm UTC](https://discuss.ray.io/t/question-about-tune-stopping-condition-with-pbt/9447/7 "2023-02-21T21:25:11Z")

</div>

Thanks ! It’s indeed quite useful for comparisons =)