# Error occurs when call save\_checkpoint

**URL:** <https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790>\
**Category:** Ray Train\
**Created:** [January 26, 2022, 4:02am UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790 "2022-01-26T04:02:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tangcc1127](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tangcc1127/32/2009_2.png) [@tangcc1127](https://discuss.ray.io/u/tangcc1127)\
**Post date:** [January 26, 2022, 4:02am UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790/1 "2022-01-26T04:02:17Z")

</div>

when I add `train.save_checkpoint(epoch, model)` in `train_func` of the scripts `train_fasion_mnist_example`, error occurs:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/0/052b53f1bb0b08626ef662da605a602b46109a83.png)

RuntimeError: Some workers returned results while others didn’t. Make sure that `train.report()` and `train.checkpoint()` are called the same number of times on all workers.

`train.save_checkpoint` follows `train.report`, and `train.report` did not raise error, but `train.save_checkpoint` did.

Did I call the method right? Or is there any other configuration I should notice?

---

<div class="post-metadata">

**Author:** ![amogkam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/amogkam/32/17_2.png) [@amogkam](https://discuss.ray.io/u/amogkam)\
**Post date:** [January 26, 2022, 4:05am UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790/2 "2022-01-26T04:05:34Z")

</div>

Hmm this usage looks right to me. How many workers are you training with? Are you using spot instances or is there anything that can cause any of the workers to fail?

Also, do you mind sharing the full stack trace? Thanks!

---

<div class="post-metadata">

**Author:** ![tangcc1127](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tangcc1127/32/2009_2.png) [@tangcc1127](https://discuss.ray.io/u/tangcc1127)\
**Post date:** [January 26, 2022, 4:34am UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790/3 "2022-01-26T04:34:04Z")

</div>

![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/b/be7db26375b8d587348fb4d325b2655122d5527c.png)

There’s the full stack trace.

Only if I add `save_checkpoint` this error will occur with `num_workers = 2`

---

<div class="post-metadata">

**Author:** ![tangcc1127](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tangcc1127/32/2009_2.png) [@tangcc1127](https://discuss.ray.io/u/tangcc1127)\
**Post date:** [January 28, 2022, 3:05am UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790/4 "2022-01-28T03:05:06Z")

</div>

Problem solved. There’s an pickle error when saving checkpoint. When I move the define of `NeuralNetwork` to another file and import it, everything works fine.

---

<div class="post-metadata">

**Author:** ![jikuaij](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jikuaij/32/2248_2.png) [@jikuaij](https://discuss.ray.io/u/jikuaij)\
**Post date:** [March 7, 2022, 12:02pm UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790/5 "2022-03-07T12:02:51Z")

</div>

how to get solving this problem, I had same problem plz send to your solution when see it

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [March 7, 2022, 10:37pm UTC](https://discuss.ray.io/t/error-occurs-when-call-save-checkpoint/4790/6 "2022-03-07T22:37:40Z")

</div>

Hey @jikuaij,

Can you share the code you are running and the output traceback?