# Saving and restoring a trial state with TrialScheduler

**URL:** <https://discuss.ray.io/t/saving-and-restoring-a-trial-state-with-trialscheduler/12449>\
**Category:** Uncategorized\
**Created:** [October 13, 2023, 5:28pm UTC](https://discuss.ray.io/t/saving-and-restoring-a-trial-state-with-trialscheduler/12449 "2023-10-13T17:28:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![caesar025](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/caesar025/32/2058_2.png) [@caesar025](https://discuss.ray.io/u/caesar025)\
**Post date:** [October 13, 2023, 5:28pm UTC](https://discuss.ray.io/t/saving-and-restoring-a-trial-state-with-trialscheduler/12449/1 "2023-10-13T17:28:35Z")

</div>

I am trying to write a custom scheduler that pauses trials at some point and then continues them at a later time. Currently the scheduler reruns the paused trials from the beginning, how can I implement this behaviour? In the example of the [Trial\_Scheduler class](https://github.com/ray-project/ray/blob/master/python/ray/tune/schedulers/trial_scheduler.py) there is no example on how to use the save/restore functions.

Abstracting the TrialScheduler class, my code looks like this:

```auto
def choose_trial_to_run(self, tune_controller: "TuneController") -> Optional[Trial]:
    for trial in tune_controller.get_trials():
        if trial.status == Trial.PAUSED and condition == True:
            # restore and return trial

def on_trial_result(self, tune_controller: "TuneController", trial: Trial, result: Dict) -> str: 
    if some_other_condition == True:
        # save trial
        return TrialScheduler.PAUSE

```

---

<div class="post-metadata">

**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\
**Post date:** [October 23, 2023, 7:01pm UTC](https://discuss.ray.io/t/saving-and-restoring-a-trial-state-with-trialscheduler/12449/2 "2023-10-23T19:01:55Z")

</div>

Hi @caesar025,

All you need to do is implement checkpoint saving/restoring in your training function. This way, when a trial “unpauses”, it’ll pull the latest checkpoint that was saved to restore the training state. Make sure that the frequency at which you checkpoint is at least as often as how frequently you pause – otherwise you’ll keep restarting from scratch without getting a chance to checkpoint.

See here:

[https://docs.ray.io/en/releases-2.7.1/train/user-guides/checkpoints.html#saving-checkpoints-during-training](https://docs.ray.io/en/releases-2.7.1/train/user-guides/checkpoints.html#saving-checkpoints-during-training)

---

<div class="post-metadata">

**Author:** ![caesar025](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/caesar025/32/2058_2.png) [@caesar025](https://discuss.ray.io/u/caesar025)\
**Post date:** [October 24, 2023, 11:08am UTC](https://discuss.ray.io/t/saving-and-restoring-a-trial-state-with-trialscheduler/12449/3 "2023-10-24T11:08:50Z")

</div>

Ok, then the save/restore functions from the Trial Scheduler class have nothing to do with saving and restoring trials?

---

<div class="post-metadata">

**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\
**Post date:** [October 24, 2023, 4:39pm UTC](https://discuss.ray.io/t/saving-and-restoring-a-trial-state-with-trialscheduler/12449/4 "2023-10-24T16:39:59Z")

</div>

Yes, the `save` and `restore` on a trial scheduler refers to saving the state of the scheduler, rather than the state of individual trials.