Ray Tune on GCP cluster: checkpoint not found after successful sync down

RickLan · April 22, 2021, 6:40am

With a cluster manually setup, I also see that last checkpoint are missing.

Your setup saves intermediate checkpoints. Maybe that’s what I should try.

May I ask which version of Ray are you running?

Topic		Replies	Views
Using Ray Tune in a Ray Cluster, checkpoints not synced back to source node Ray Tune	9	922	April 22, 2021
Sync down not happening when using cloud checkpointing Ray Tune	6	538	July 23, 2021
Cannot find checkpoint when gpus_per_trial > 0 Ray Tune	8	554	February 28, 2023
Rsync Error when using Ray Tune Ray Tune	4	1030	December 14, 2020
Ray Tune x SLURM - Problem with checkpoints Ray Libraries (Data, Train, Tune, Serve)	5	294	March 15, 2023