Hello,
My Problem
I am attempting to use a LakeFS S3 bucket as a shared storage location for my ray workers during a ray Tune job. I am running ray 2.34.0. I am using a private endpoint url, so I have the following environment variables set: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_ENDPOINT_URL. I have verified that these credentials are correct by both pushing and pulling data from an s3 bucket on the instance using awscli and boto3.
However, when I configure my ray.tune.RunConfig to point to the bucket via ray.train.RunConfig(name="<experiment_name>", storage_path=s3://<my_bucket>/main)
(where main is the main branch in my lakefs repo), my tuner initialization fails with the following error:
ERROR:root:When creating key 'main/' in bucket '<my_bucket>': AWS Error RESOURCE_NOT_FOUND during PutObject operation: No response body.
My Investigation
In an attempt to figure out the issue, I dove into the ray source code and examined the ray.train._internal_storage.StorageContext object, which seems to be the object that handles the storage parameters that are passed in to the RayConfig. Under the hood, it looks like StorageContext uses a pyarrow.fs.FileSystem to reference the provided storage uri. StorageContext’s constructor seems to create a validation file and attempt to push it to the provided uri, which involves calling pyarrow.fs.FileSystem.create_dir(experiment_fs_path) (where experiment_fs_path is the combination of the original storage uri and the experiment name). Its my understanding that this function should create the path in the bucket for the file upload, but its on this call to pyarrow.fs.FileSystem.create_dir that the error is thrown.
Any help here would be greatly appreciated!