I am running into issue in creating checkpoint to s3 the same code works fine for nfs storage, Even on S3 It successfully creates an a directory in s3 bucket later for file creation it runs into AWS Error (HTTP status 400) during upload part operation of run_config of ray.
Below is the complete error.
/arrow/cpp/src/arrow/io/interfaces.cc:288: Error ignored when destroying file of type N5arrow2fs12_GLOBAL__N_118ObjectOutputStreamE: IOError: When uploading part for key ‘vss31-ptl-mnist-example/.validate_storage_marker’ in bucket ‘mlcheckpoints’: AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: InvalidPart Message: One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part’s entity tag.