# How to continue errored out tune.run

**URL:** <https://discuss.ray.io/t/how-to-continue-errored-out-tune-run/5600>\
**Category:** Ray Tune\
**Created:** [March 29, 2022, 9:36pm UTC](https://discuss.ray.io/t/how-to-continue-errored-out-tune-run/5600 "2022-03-29T21:36:18Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andres\_Kaver](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andres_kaver/32/2335_2.png) [@Andres\_Kaver](https://discuss.ray.io/u/Andres_Kaver)\
**Post date:** [March 29, 2022, 9:36pm UTC](https://discuss.ray.io/t/how-to-continue-errored-out-tune-run/5600/1 "2022-03-29T21:36:18Z")

</div>

- High: It blocks me to complete my task.

I have a weird bug with my RTX3090 - it throws CUDA OOM error after about 3 hours of training. Only way so far to fix it, is to stop tune training manually and then restart with resume=True. If i set up max\_failures=0 (to get it to stop automatically) then resume will not do anything as all the experiments are in trial.ERROR state.

Basically what i need to do is to reset trial state to PAUSED and then start (resume) run from last checkpoints.

How can I change all the trials to different state before calling tune.run?

I can do it manually in experiment directory in experiment\_state-.json - but is there some other way, via code?

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [April 4, 2022, 6:13pm UTC](https://discuss.ray.io/t/how-to-continue-errored-out-tune-run/5600/2 "2022-04-04T18:13:53Z")

</div>

Hi @Andres_Kaver, there is currently no great way to continue training after it gracefully finished, but we’re looking into this! For the original problem, can you share a bit of your code, e.g. the configuration and the call to `tune.run()`? Do multiple trials access the same GPU or is it just one trial that increases GPU memory utilization over time? If so that sounds like a memory leak, but if it’s more than one trial it could also be a bookkeeping issue on the Ray side.