Note for others following: Based on this post you can build a custom context function to wrap the trainer for loop and this could automatically save a checkpoint on model cleanup.
philipjb
12
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| BOHB restarting from scratch rather than last checkpoint with GCP syncing | 5 | 423 | March 25, 2022 | |
| Why ray tune restarts my trainings? | 7 | 550 | June 27, 2022 | |
| Checkpointing with distributed training | 14 | 1055 | April 20, 2021 | |
| Correct way of resuming trials | 4 | 2476 | May 20, 2022 | |
| [Tune] How to turn off checkpointing for testing | 20 | 3472 | April 18, 2023 |