Caught sync error: Sync process failed: Connect timeout on endpoint URL

Ray Libraries (Data, Train, Tune, Serve)
1

This is how I’ve configured ray tune.Tuner

tuner = tune.Tuner(tune.with_resources(...), tune_config=tune.TuneConfig(...), run_config=air.RungConfig(local_dir=os.path.join(os.getcwd(), 'raytune_folder/', sync_config=tune.SyncConfig(upload_dir='s3://raytune/test/', syncer='auto', sync_artifacts=True, sync_on_checkpoint=True,), checkpoint_config=ray.air.CheckpointConfig(num_to_keep=5,...), progress_reporter=reporter, vebose=1, log_to_file=True))

This outputs error as mentioned in the title. Weird thing is it worked with python3.7 however starting to throw this error after migrating to python3.8 and on the error message “Connect timeout on endpoint URL : https://s3.us-east-2.amazonaws.com/raytune/test/experiment-state-2023-10-16_09-41-01.json” the region is wrong.

2

@Haneul_Kim Could you try this on Ray 2.7? There should only be minimal changes, and the storage backend has been revamped since the ray version you’re using.

from ray import train, tune

tuner = tune.Tuner(
    tune.with_resources(...),
    run_config=train.RunConfig(
        local_dir=os.path.join(os.getcwd(), 'raytune_folder/'),
        storage_path="s3://raytune/test/", 
        # sync_config=train.SyncConfig(sync_artifacts=True),   # optional
    ),
    ...
)

If the region is still an issue, try setting the env variable AWS_DEFAULT_REGION.

See here for more info: Configuring Persistent Storage — Ray 3.0.0.dev0