How to resume training from a checkpoint

[starkj]

@Finebouche I had the same questions a few months ago. I found it odd that Ray Tune doesn’t provide a straightforward way to start a new Tune job from an existing Tune checkpoint. You have to go through some gyrations. I eventually learned how to use a callback to restore just the policy weights from a checkpoint and proceed from there. Of course, that starts over with no optimizer params & other training loop counters, etc. But the raw NN itself can be moved forward. For details, see Tune as part of curriculum training - #14 by gjoliver. It is a bit of a circuitous discussion for a while, but about half way down you see talk of the callbacks.