Error occurs when call save_checkpoint

Ray Train
#1

when I add train.save_checkpoint(epoch, model) in train_func of the scripts train_fasion_mnist_example, error occurs:

image
image687×575 55.5 KB

RuntimeError: Some workers returned results while others didn’t. Make sure that train.report() and train.checkpoint() are called the same number of times on all workers.

train.save_checkpoint follows train.report, and train.report did not raise error, but train.save_checkpoint did.

Did I call the method right? Or is there any other configuration I should notice?

#2

Hmm this usage looks right to me. How many workers are you training with? Are you using spot instances or is there anything that can cause any of the workers to fail?

Also, do you mind sharing the full stack trace? Thanks!

#3

image
image961×690 120 KB

There’s the full stack trace.

Only if I add save_checkpoint this error will occur with num_workers = 2

#4

Problem solved. There’s an pickle error when saving checkpoint. When I move the define of NeuralNetwork to another file and import it, everything works fine.

#5

how to get solving this problem, I had same problem plz send to your solution when see it

#6

Hey @jikuaij,

Can you share the code you are running and the output traceback?