How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
I’m recently upgrading my projects from 1.13.x to 2.2.0 and I noticed that sync up to s3 is somehow working differently.
- 1.13.X: everything under experiment’s
local_dirwill be synced up to S3.
- 2.2.0: My observation is that only checkpoint folders generated on workers node will be synced up to s3 and files on head node will be ignored. And terminal output during experiment running also suggests that sync up only happening on the worker nodes.
Is this expected behavior? How can I force syncing up on head node?