How to access Amazon S3

Hi @Gil_Vernik! If you set your AWS credentials via the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, Datasets should use those credentials without any code changes.

If this environment variable method isn’t agreeable, you can pass ray.data.read_binary_files() an Arrow S3FileSystem instance containing your AWS credentials (see the .read_binary_files() API).

1 Like