# Can't save Checkpoint wenn using Tensorflow and PBT

**URL:** <https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419>\
**Category:** Ray Tune\
**Created:** [January 11, 2021, 4:36pm UTC](https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419 "2021-01-11T16:36:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![KaiLiu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kailiu/32/233_2.png) [@KaiLiu](https://discuss.ray.io/u/KaiLiu)\
**Post date:** [January 11, 2021, 4:36pm UTC](https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419/1 "2021-01-11T16:36:28Z")

</div>

Hallo everyone, I was using ray tune PBT to tune my model, but I can’t find the saved model in checkpoints Dokumente. At the end of traing I can get the best config but I can’t get the best model. And i will get error like this:

> (pid=27000) 2021-01-11 17:16:43.197133: W tensorflow/core/framework/op\_kernel.cc:1767] OP\_REQUIRES failed at save\_restore\_v2\_ops.cc:109  
> : Not found: Failed to create a NewWriteableFile: D:\probe\pbt\_checkpoint\pbt\_test\MLPmodel\_39793\_00000\_0\_af\_0=0,af\_1=2,af\_2=1,af\_3=0,af\_4=2,af\_5=2,af\_6=2,af\_7=1,af\_output=3,batchsize=849,num\_layers=5,units\_0=498,\_2021-01-11\_17-15-41\variables\variables\_temp\_62b4641374534df4bd63c5ecfd5991b3/part-00000-of-00001.data-00000-of-00001.tempstate15293471527192116649 : ϵͳ�Ҳ���ָ����·����  
> (pid=27000) ; No such process  
> 2021-01-11 17:16:43,978 ERROR worker.py:980 – Possible unhandled error from worker: ray::MLPmodel.save\_to\_object() (pid=27000, ip=172.16.1.32)  
> File “python\ray\_raylet.pyx”, line 463, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 415, in ray.\_raylet.execute\_task.function\_executor  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\ray\function\_manager.py”, line 556, in actor\_method\_executor  
> return method(\_\_ray\_actor, \*args, \*\*kwargs)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\ray\tune\trainable.py”, line 295, in save\_to\_object  
> checkpoint\_path = self.save(tmpdir)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\ray\tune\trainable.py”, line 278, in save  
> checkpoint = self.save\_checkpoint(checkpoint\_dir)  
> File “d:/Probe/PBT/PBT\_probe.py”, line 94, in save\_checkpoint  
> self.model.save(file\_path)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\keras\engine\training.py”, line 1979, in save  
> signatures, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\keras\saving\save.py”, line 134, in save\_model  
> signatures, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\keras\saving\saved\_model\save.py”, line 80, in save  
> save\_lib.save(model, filepath, signatures, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\saved\_model\save.py”, line 985, in save  
> options=ckpt\_options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\tracking\util.py”, line 1200, in save  
> file\_prefix\_tensor, object\_graph\_tensor, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\tracking\util.py”, line 1145, in \_save\_cached\_when\_graph\_building  
> save\_op = saver.save(file\_prefix, options=options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\saving\functional\_saver.py”, line 295, in save  
> return save\_fn()  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\saving\functional\_saver.py”, line 269, in save\_fn  
> sharded\_saves.append(saver.save(shard\_prefix, options))  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\saving\functional\_saver.py”, line 78, in save  
> return io\_ops.save\_v2(file\_prefix, tensor\_names, tensor\_slices, tensors)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\ops\gen\_io\_ops.py”, line 1731, in save\_v2  
> ctx=\_ctx)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\ops\gen\_io\_ops.py”, line 1751, in save\_v2\_eager\_fallback  
> ctx=ctx, name=name)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\eager\execute.py”, line 60, in quick\_execute  
> inputs, attrs, num\_outputs)  
> tensorflow.python.framework.errors\_impl.NotFoundError: Failed to create a directory: D:\probe\pbt\_checkpoint\pbt\_test\MLPmodel\_39793\_00000\_0\_af\_0=0,af\_1=2,af\_2=1,af\_3=0,af\_4=2,af\_5=2,af\_6=2,af\_7=1,af\_output=3,batchsize=849,num\_layers=5,units\_0=498,\_2021-01-11\_17-15-41\tmpulblmjabsave\_to\_object\checkpoint\_2/model\variables\variables\_temp\_c123b60a74554ae3b4da1f882fa4089b; No such file or directory [Op:SaveV2]  
> 2021-01-11 17:16:49,136 ERROR worker.py:980 – Possible unhandled error from worker: ray::MLPmodel.stop() (pid=27000, ip=172.16.1.32)  
> File “python\ray\_raylet.pyx”, line 463, in ray.\_raylet.execute\_task  
> File “python\ray\_raylet.pyx”, line 415, in ray.\_raylet.execute\_task.function\_executor  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\ray\function\_manager.py”, line 556, in actor\_method\_executor  
> return method(\_\_ray\_actor, \*args, \*\*kwargs)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\ray\tune\trainable.py”, line 512, in stop  
> self.cleanup()  
> File “d:/Probe/PBT/PBT\_probe.py”, line 102, in cleanup  
> saved\_path = self.model.save(self.logdir)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\keras\engine\training.py”, line 1979, in save  
> signatures, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\keras\saving\save.py”, line 134, in save\_model  
> signatures, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\keras\saving\saved\_model\save.py”, line 80, in save  
> save\_lib.save(model, filepath, signatures, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\saved\_model\save.py”, line 985, in save  
> options=ckpt\_options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\tracking\util.py”, line 1200, in save  
> file\_prefix\_tensor, object\_graph\_tensor, options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\tracking\util.py”, line 1145, in \_save\_cached\_when\_graph\_building  
> save\_op = saver.save(file\_prefix, options=options)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\saving\functional\_saver.py”, line 295, in save  
> return save\_fn()  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\saving\functional\_saver.py”, line 269, in save\_fn  
> sharded\_saves.append(saver.save(shard\_prefix, options))  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\training\saving\functional\_saver.py”, line 78, in save  
> return io\_ops.save\_v2(file\_prefix, tensor\_names, tensor\_slices, tensors)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\ops\gen\_io\_ops.py”, line 1731, in save\_v2  
> ctx=\_ctx)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\ops\gen\_io\_ops.py”, line 1751, in save\_v2\_eager\_fallback  
> ctx=ctx, name=name)  
> File “D:\anaconda3\envs\BA\_37\lib\site-packages\tensorflow\python\eager\execute.py”, line 60, in quick\_execute  
> inputs, attrs, num\_outputs)  
> UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xd5 in position 355: invalid continuation byte

and I have used this code to save checkpoint

> def save\_checkpoint(self, checkpoint\_dir):
> 
> ```
> file_path = checkpoint_dir + "/model"
> 
> self.model.save(file_path)
> 
> return file_path
> 
> def load_checkpoint(self, path):
> 
> del self.model
> 
> self.model = load_model(path)
> 
> ```

Hier is my ray.run function:

> pbt = PopulationBasedTraining(
> 
> ```
> time_attr="training_iteration",
> 
> perturbation_interval=2,
> 
> hyperparam_mutations=mutationspace)
> 
> results = tune.run(
> 
> MLPmodel,
> 
> name="pbt_test",
> 
> local_dir=os.path.normpath('D:/probe/pbt_checkpoint/'),
> 
> scheduler=pbt,
> 
> metric="msle",
> 
> mode="min",
> 
> reuse_actors=True,
> 
> resources_per_trial={
> 
> "cpu": 3,
> 
> "gpu": 1
> 
> },
> 
> stop={"training_iteration": 4},
> 
> num_samples=2,
> 
> config=searchspace,
> 
> )
> 
> ```

Any suggestions why this might happend and how to fix it?

Thank you

---

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [January 11, 2021, 4:45pm UTC](https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419/2 "2021-01-11T16:45:52Z")

</div>

You can add for eaxample lines:

```auto
results = tune.run(

```

> ```
> keep_checkpoints_num=3,
> checkpoint_freq=3,
> checkpoint_at_end=True,
> 
> ```

keep\_checkpoints\_num - save the last 3 models (earlier models are automatically deleted)  
checkpoint\_freq - save model for every 3 iterations  
checkpoint\_at\_end - keep the last checkpoint

look for in the ‘ray\_results’ the directories with namedes: ‘checkpoint\_x’ - where x is iteration number

Peter

---

<div class="post-metadata">

**Author:** ![KaiLiu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kailiu/32/233_2.png) [@KaiLiu](https://discuss.ray.io/u/KaiLiu)\
**Post date:** [January 11, 2021, 7:34pm UTC](https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419/3 "2021-01-11T19:34:44Z")

</div>

Hi Peter,  
Thanks for your advice! I have followed your advice, but I can’t see the model under ‘checkpoint\_x/model’, and I got the error log as follows:

> ray.exceptions.RayTaskError(NotFoundError): e[36mray::MLPmodel.save()e[39m (pid=26136, ip=172.16.1.32)  
> tensorflow.python.framework.errors\_impl.NotFoundError: Failed to create a directory: D:\probe\pbt\_checkpoint\pbt\_test\MLPmodel\_9a61d\_00001\_1\_af\_0=1,af\_1=2,af\_2=2,af\_3=2,af\_4=0,af\_5=1,af\_6=2,af\_7=1,af\_output=3,batchsize=1006,num\_layers=4,units\_0=62,\_2021-01-11\_20-10-53\checkpoint\_3/model\variables\variables\_temp\_fc9757f57fd84106aebf0d0346f99f22; No such file or directory [Op:SaveV2]  
> maybe the name for checkpoint is too long for tensorflow to save, so I can’t save the model, do you have any idea?

---

<div class="post-metadata">

**Author:** ![Peter\_Pirog](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/peter_pirog/32/234_2.png) [@Peter\_Pirog](https://discuss.ray.io/u/Peter_Pirog)\
**Post date:** [January 11, 2021, 9:57pm UTC](https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419/4 "2021-01-11T21:57:01Z")

</div>

![checkpoint_directory](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/576dbd75ac2eadddbd43620acc1cb7d5d9d500b3.png)

This is checkpoint directory. Simple code if you want to use PPO trainer on typical gym environment, (it’s not perfect but works):

> analysis = tune.run(  
> run\_or\_experiment=“PPO”, # check if your environment is continous or discreete before choosing training algorithm  
> scheduler=asha\_scheduler,  
> keep\_checkpoints\_num=3,  
> checkpoint\_freq=3,  
> checkpoint\_at\_end=True,  
> stop={“episode\_reward\_mean”: 300}, # stop training if this value is reached episode\_reward\_mean  
> mode=‘max’, # find maximum vale as a target  
> reuse\_actors=True,  
> config=config,  
> verbose=3, #0 = silent, 1 = only status updates, 2 = status and brief trial results, 3 = status and detailed trial results. Defaults to 3  
> )
> 
> checkpoints = analysis.get\_trial\_checkpoints\_paths(  
> trial=analysis.get\_best\_trial(“episode\_reward\_mean”),  
> metric=“episode\_reward\_mean”)
> 
> print(‘checkpoints=’, checkpoints)  
> checkpoint\_path, reward = checkpoints[0]  
> print(‘checkpoint\_path=’, checkpoint\_path)
> 
> config = {  
> “env”: “CartPole-v0”,  
> “num\_gpus”: 0,  
> “num\_workers”: 1,  
> “framework”: “tf2”,  
> }
> 
> agent = ppo.PPOTrainer(config=config, env=“CartPole-v0”)  
> agent.restore(checkpoint\_path)
> 
> print(‘agent=’, agent)
> 
> ############## TYPICAL GYM ENV ##########################  
> import gym  
> env = gym.make(“CartPole-v0”)
> 
> episode\_reward = 0  
> done = False  
> obs = env.reset()  
> while not done:  
> action = agent.compute\_action(obs)  
> obs, reward, done, info = env.step(action)  
> episode\_reward += reward  
> print(‘episode\_reward=’, episode\_reward)

This path works for me:  
/home/peterpirog/PycharmProjects/Ray\_tests/ray\_results/PPO/PPO\_BipedalWalkerHardcore-v3\_eba13\_00000\_0\_2021-01-10\_16-43-09/checkpoint\_1326

Maybe check if paths are correct:

 ![slashes](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/f78b93f5c71bdbe481ce3c3937eb7ba63d6c49b3.png)

---

<div class="post-metadata">

**Author:** ![KaiLiu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kailiu/32/233_2.png) [@KaiLiu](https://discuss.ray.io/u/KaiLiu)\
**Post date:** [January 12, 2021, 1:06pm UTC](https://discuss.ray.io/t/cant-save-checkpoint-wenn-using-tensorflow-and-pbt/419/5 "2021-01-12T13:06:54Z")

</div>

Hallo Peter, thanks for your advice. I am using Ray tune to tune the number of hidden layer of a MLP Network, hier is my script to tune the model, could you please check this script?

> [GitHub - LiuDaniu1997/tune-with-ray](https://github.com/LiuDaniu1997/tune-with-ray)

Is it possible to tune the number of hidden layer with PBT or any other methode in Ray?