Hi @Gil_Vernik! If you set your AWS credentials via the AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables, Datasets should use those credentials without any code changes.
If this environment variable method isn’t agreeable, you can pass ray.data.read_binary_files()
an Arrow S3FileSystem
instance containing your AWS credentials (see the .read_binary_files()
API).