How to configure Ray to access Amazon S3

Ray Libraries (Data, Train, Tune, Serve) Ray Data
1

There are various examples how Ray can read and write data from Amazon S3, for example

ds.repartition(1).write_json("s3://bucket/inference-results")

How to configure Ray with S3 credentials? I don’t run Ray in AWS, I run it locally on my laptop and want to read data from my Amazon S3 and also write there.

Thanks

2

What’s the error message that you get?

You should be able to get this working with aws configure and making sure you have read/write permissions to your S3 bucket.

3

If i use Ray cluster, with few worker nodes, then i also need to run aws configure as well?
I thought there is some configuration option in Ray itself to configure S3 access…

Thanks

4

Yeah, you need to either run aws configure or set the relevant AWS environment variables to access S3.

cc @Clark_Zinzow

5

@rliaw Answered in this thread in Core posted by the same user: How to access Amazon S3 - #2 by Clark_Zinzow

1 Like