# Sync down not happening when using cloud checkpointing **URL:** \ **Category:** Ray Tune\ **Created:** [July 20, 2021, 1:26pm UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940 "2021-07-20T13:26:19Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![iamhatesz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/iamhatesz/32/778_2.png) [@iamhatesz](https://discuss.ray.io/u/iamhatesz)\ **Post date:** [July 20, 2021, 1:26pm UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/1 "2021-07-20T13:26:20Z")
Hi! I am working on a pipeline for running a Tune+RLlib training on AWS. Ray Cluster is being managed by Kubernetes ray-operator. Now, for a job of training a PPO agent, there is a ray-head and one ray-worker pods being created. I used a following `sync_config`: ```python sync = tune.SyncConfig( upload_dir=f"s3://my-bucket", sync_to_driver=False, sync_to_cloud="aws s3 sync {source} {target}", node_sync_period=10, cloud_sync_period=10, ) ``` and wrapped trainable with `tune.durable`: ```python tune.tune.run( tune.durable("PPO"), # ... checkpoint_freq=1, checkpoint_at_end=True, keep_checkpoints_num=3, checkpoint_score_attr="episode_reward_mean", # ... sync_config=sync, ) ``` The training works, but the checkpoints aren’t synced. They are present on worker pod, but not on the head node. In the logs I’ve observed a lot of logs like this one: ```auto (pid=388)e[0m 2021-07-19 17:28:29,359 ERROR trial_runner.py:915 -- Trial PPO_WebEnv_7cb70_00000: Error handling checkpoint /root/ray_results/short/PPO_WebEnv_7cb70_00000_0_max_nodes=150,visited_states=True_2021-07-19_17-25-02/checkpoint_000001/checkpoint-1 (pid=388)e[0m Traceback (most recent call last): (pid=388)e[0m File "/root/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 907, in _process_trial_save (pid=388)e[0m self._callbacks.on_checkpoint( (pid=388)e[0m File "/root/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/tune/callback.py", line 216, in on_checkpoint (pid=388)e[0m callback.on_checkpoint(**info) (pid=388)e[0m File "/root/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/tune/syncer.py", line 455, in on_checkpoint (pid=388)e[0m self._sync_trial_checkpoint(trial, checkpoint) (pid=388)e[0m File "/root/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/tune/syncer.py", line 428, in _sync_trial_checkpoint (pid=388)e[0m raise TuneError("Trial {}: Checkpoint path {} not " (pid=388)e[0m ray.tune.error.TuneError: Trial PPO_WebEnv_7cb70_00000: Checkpoint path /root/ray_results/short/PPO_WebEnv_7cb70_00000_0_max_nodes=150,visited_states=True_2021-07-19_17-25-02/checkpoint_000001/checkpoint-1 not found after successful sync down. ``` I’ve used custom `sync_to_cloud` function just to avoid passing `--only-show-errors` flag to `aws s3 sync` command. In the logs I see also info about syncing up: ```auto upload: ../root/ray_results/short/PPO_WebEnv_7cb70_00000_0_max_nodes=150,visited_states=True_2021-07-19_17-25-02/events.out.tfevents.1626708470.ray-cluster-training-ray-head-type-z24cp to s3://alan-system-nonproduction/training/cb9a5922/short/PPO_WebEnv_7cb70_00000_0_max_nodes=150,visited_states=True_2021-07-19_17-25-02/events.out.tfevents.1626708470.ray-cluster-training-ray-head-type-z24cp ``` but no info about syncing it down. Also, there is no information about syncing up checkpoints (just TensorBoard logs, experiment state, etc.). At the same time, if I use AWS CLI, the checkpoints are present in the bucket. I’ve also verified that all pods are capable of syncing with this bucket. I’ve tested this on Ray 1.4.1 and wheels from this commit: [[tune] Pass custom `sync_to_cloud` templates to durable trainables (#… · ray-project/ray@4178655 · GitHub](https://github.com/ray-project/ray/commit/4178655ba70838734d8f5b0ca9ff885f3f4f5c48) Do you have any idea what might be wrong? ---
**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\ **Post date:** [July 20, 2021, 2:08pm UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/2 "2021-07-20T14:08:59Z")
Can you try setting `sync_on_checkpoint=False` in the SyncConfig? Using a durable trainable should not require syncing to the head node/pod at all, since all checkpoints should live on cloud storage. But I saw this error earlier today, so if it persists, I’m happy to look more into it. ---
**Author:** ![iamhatesz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/iamhatesz/32/778_2.png) [@iamhatesz](https://discuss.ray.io/u/iamhatesz)\ **Post date:** [July 20, 2021, 3:03pm UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/3 "2021-07-20T15:03:09Z")
Hey @kai. Indeed, with `sync_on_checkpoint=False` the error disappeared, but the checkpoints are still not present on the head pod. Is this a valid behavior for Tune that checkpoints are stored only on one pod, in S3, but not synced down to the other pods? ```auto root@ray-cluster-training-ray-worker-type-c5lxd:~/ray_results/short/PPO_WebEnv_7c88c_00000_0_max_nodes=150,visited_states=True_2021-07-20_17-02-22# ls -l total 0 drwxr-xr-x 2 root root 82 Jul 20 17:05 checkpoint_000001 drwxr-xr-x 2 root root 82 Jul 20 17:06 checkpoint_000002 ``` ```auto root@ray-cluster-training-ray-head-type-ds9jz:~/ray_results/short/PPO_WebEnv_7c88c_00000_0_max_nodes=150,visited_states=True_2021-07-20_17-02-22# ls -l total 84 -rw-r--r-- 1 root root 21700 Jul 20 17:06 events.out.tfevents.1626793495.ray-cluster-training-ray-head-type-ds9jz -rw-r--r-- 1 root root 5268 Jul 20 17:04 params.json -rw-r--r-- 1 root root 4524 Jul 20 17:04 params.pkl -rw-r--r-- 1 root root 11897 Jul 20 17:06 progress.csv -rw-r--r-- 1 root root 31000 Jul 20 17:06 result.json ``` ---
**Author:** ![iamhatesz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/iamhatesz/32/778_2.png) [@iamhatesz](https://discuss.ray.io/u/iamhatesz)\ **Post date:** [July 23, 2021, 10:26am UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/4 "2021-07-23T10:26:51Z")
@kai did you have a chance to take a look at this? ---
**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\ **Post date:** [July 23, 2021, 10:53am UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/5 "2021-07-23T10:53:41Z")
Sorry for the delay. If using a durable trainable, checkpoints will only be stored on the local nodes that executed the trials and on S3. Syncing to the driver is currently not supported for durable trainables - this is mostly because we could end up with data inconsistencies (we don’t want to continuously sync new checkpoints). So in the case that you’d want to have access to the checkpoints locally after training, you’d have to fetch them from s3 manually. ---
**Author:** ![iamhatesz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/iamhatesz/32/778_2.png) [@iamhatesz](https://discuss.ray.io/u/iamhatesz)\ **Post date:** [July 23, 2021, 11:01am UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/6 "2021-07-23T11:01:26Z")
Thanks for the answer. Still, there is one thing that bothers me - in case of a pod failure (e.g. when using AWS Spot instances), how does Tune recover failed workers? Is it going to spawn new workers and sync checkpoints there? ---
**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\ **Post date:** [July 23, 2021, 2:27pm UTC](https://discuss.ray.io/t/sync-down-not-happening-when-using-cloud-checkpointing/2940/7 "2021-07-23T14:27:46Z")
Yes, this is exactly what should happen!