# Tuner cannot restore the checkpoints! **URL:** \ **Category:** Ray Tune\ **Created:** [June 1, 2023, 9:45pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880 "2023-06-01T21:45:00Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![deepgravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/deepgravity/32/2039_2.png) [@deepgravity](https://discuss.ray.io/u/deepgravity)\ **Post date:** [June 1, 2023, 9:45pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/1 "2023-06-01T21:45:00Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. Hi all, I am using `Tuner` and `Tuner.fit()` to train a `PPO` agent on my custom env. I successfully train the agent for some time without any errors. Now I would like to restore the checkpoints for further training; however, it gives me the following error ```auto *** RuntimeError: Could not find Tuner state in restore directory. Did you passthe correct path (including experiment directory?) ``` This is how my checkpoint directory looks like: ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/a/a20193b98753c9cb522639ea2b0664ba30226334.png) This is my code: ```auto tuner = tune.Tuner( "PPO", run_config=run_config, param_space=param_space, ) chkpt_path = "/home/PPO/PPO_MasterEnv_214c3_00000_0_2023-06-01_15-13-42/checkpoint_001300/" tuner.restore(chkpt_path) results = tuner.fit() ``` What is more strange is that I can restore the checkpoint with the `train` and use the model for prediction, or further training like this: ```auto algo =param_space.build() chkpt_path = "/home/PPO/PPO_MasterEnv_214c3_00000_0_2023-06-01_15-13-42/checkpoint_001300/" algo.restore(chkpt_path) self.algo.train() ``` But I would like to use the `Tuner`, and this gives me that error! Nor the [`Rllib` documentation](https://docs.ray.io/en/latest/rllib/rllib-saving-and-loading-algos-and-policies.html) neither [this example](https://github.com/ray-project/ray/blob/master/rllib/examples/documentation/saving_and_loading_algos_and_policies.py) shows how to restore the checkpoints for `Tuner`. Can anyone help me to fix this? Thanks! ---
**Author:** ![yunxuanx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yunxuanx/32/4354_2.png) [@yunxuanx](https://discuss.ray.io/u/yunxuanx)\ **Post date:** [June 8, 2023, 3:36am UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/2 "2023-06-08T03:36:06Z")
Hi @deepgravity , to restore a tuner, you have to pass the experiment dir path (`/home/PPO`), instead of the checkpoint path. ---
**Author:** ![deepgravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/deepgravity/32/2039_2.png) [@deepgravity](https://discuss.ray.io/u/deepgravity)\ **Post date:** [June 8, 2023, 9:15am UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/3 "2023-06-08T09:15:15Z")
> [@deepgravity](#): > > ```auto > > ``` Hi @yunxuanx , thank you for your reply. Yes, I had finally figured it out! It is really strange though. If I could remember correctly, in Rax 1.x, we passed the checkpoint dir path! ---
**Author:** ![Shengchao\_Y](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shengchao_y/32/5302_2.png) [@Shengchao\_Y](https://discuss.ray.io/u/Shengchao_Y)\ **Post date:** [November 9, 2023, 1:00pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/4 "2023-11-09T13:00:16Z")
Hi @deepgravity , I restored the tuner with “/path\_to/PPO” , but the tuner starts training from scratch rather than from the last checkpoint. How have you solved this? ---
**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\ **Post date:** [November 9, 2023, 9:17pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/5 "2023-11-09T21:17:08Z")
Hey @Shengchao_Y, how are you terminating the original run? Do you force kill the experiment? ---
**Author:** ![Shengchao\_Y](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shengchao_y/32/5302_2.png) [@Shengchao\_Y](https://discuss.ray.io/u/Shengchao_Y)\ **Post date:** [November 14, 2023, 11:56am UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/6 "2023-11-14T11:56:27Z")
Hey @justinvyu , the original run was terminated due to an internal caused by a rarely happenning environment error. I would like to cuntinue the trial after fixing this error. ---
**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\ **Post date:** [November 14, 2023, 5:50pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/7 "2023-11-14T17:50:37Z")
Were any checkpoints saved before the run terminated? How long was it running for before the environment error? ---
**Author:** ![Shengchao\_Y](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shengchao_y/32/5302_2.png) [@Shengchao\_Y](https://discuss.ray.io/u/Shengchao_Y)\ **Post date:** [November 15, 2023, 8:59pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/8 "2023-11-15T20:59:59Z")
Yes, I saved checkpoints every 20 iterations. It ran for about one and half days. ---
**Author:** ![deepgravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/deepgravity/32/2039_2.png) [@deepgravity](https://discuss.ray.io/u/deepgravity)\ **Post date:** [November 16, 2023, 10:24pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/9 "2023-11-16T22:24:10Z")
I think you need to share some codes, otherwise, it would be hard to figure out a solution. ---
**Author:** ![Windy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/windy/32/5342_2.png) [@Windy](https://discuss.ray.io/u/Windy)\ **Post date:** [November 19, 2023, 3:50pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/10 "2023-11-19T15:50:40Z")
oh man your code is so helpful for me, it took me 2 hours to fix it.Thanks ---
**Author:** ![deepgravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/deepgravity/32/2039_2.png) [@deepgravity](https://discuss.ray.io/u/deepgravity)\ **Post date:** [November 20, 2023, 8:13pm UTC](https://discuss.ray.io/t/tuner-cannot-restore-the-checkpoints/10880/11 "2023-11-20T20:13:51Z")
Happy you found it helpful