When starting a run with tune.run_experiments, how do you pass a SyncConfig object to set the “upload_folder” for syncing to an S3 bucket? In prior versions the upload_dir was just set directly in the experiment spec.
Trying to get some sagemaker examples to work with Ray 1.9.2:
experiment_config["training"]["config"]["use_pytorch"] = use_pytorch
del experiment_config["training"]["config"]["framework"]
else: # if "use_pytorch" is used or no framework specified
use_pytorch = experiment_config["training"]["config"].get("use_pytorch", False)
if ray.__version__ > "0.8.5":
experiment_config["training"]["config"]["framework"] = (
"torch" if use_pytorch else "tf"
)
experiment_config["training"]["config"].pop("use_pytorch", None)
run_experiments(experiment_config)
all_workers_host_names = self.get_all_host_names()[1:]
# If distributed job, send TERMINATION_SIGNAL to all workers.
if len(all_workers_host_names) > 0:
self.sage_cluster_communicator.create_s3_signal(TERMINATION_SIGNAL)
algo = experiment_config["training"]["run"]
env_string = experiment_config["training"]["config"]["env"]
self.save_checkpoint_and_serving_model(
algorithm=algo, env_string=env_string, use_pytorch=use_pytorch
)
rliaw
February 8, 2022, 9:18am
2
I’m afraid this isn’t supported directly on Ray right now – can you file a feature request? it should be straightforward to support.