I posted this in Ray Data, but seems it less active than the core forum. Sorry for multiple posts.
There are various examples how Ray can read and write data from Amazon S3, for example
ds = ray.data.read_binary_files("s3://bucket/image-dir")
How to configure Ray with S3 credentials? I don’t run Ray in AWS, I run it locally on my laptop (just installed it with pip ) and I want to read data from my Amazon S3 and also write there.
Hi @Gil_Vernik! If you set your AWS credentials via the
AWS_SECRET_ACCESS_KEY environment variables, Datasets should use those credentials without any code changes.
If this environment variable method isn’t agreeable, you can pass
ray.data.read_binary_files() an Arrow
S3FileSystem instance containing your AWS credentials (see the
Hi, @Clark_Zinzow how to set all environment variables when I use MinIO locally, since I don’t like to pass Arrrow S3FileSystem instance to .read_binary_files API.