I can't resume training

Ray Tune
1

I can’t resume training. Say I run python egt_node_gat.py --framework torch --stop-iters 500 --stop-reward 1 and it reaches iter 500 and stops.

Now I want to keep training further so I run the script again with a higher --stop-iters, but this time with resume="PROMPT" in tune.run.

and it terminates immediately
Number of trials: 1/1 (1 TERMINATED)

Should I be doing anything differently?

2

Hi @TheExGenesis,

I think the resume option is for runs that did not complete for some reason.

F from your description it sounds like your first run terminated cleanly.

I think what you want is to to start a new run using the restore option to reload your last checkpoint from before. You will still need to increase the total stop-inters because the check is cumulative.