Unable to locate credentials with S3 Remote URIs

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi!

I have some data I’m trying to process on my worker nodes and my understanding is that I’d need to use Remote URIs because even when zipped it’s greater than 100 MB. I zipped it up and put it in S3 but I get the following error whenever Ray tries to create the working_dir:

Failed to create runtime_env for Ray client server, it is caused by:
botocore.exceptions.NoCredentialsError: Unable to locate credentials

I have set up AWS credentials, and the following runs without issue:

import io
import os
import uuid
import boto3

def list_bucket_contents(s3_resource):
    bucket_name = f"my-tmp-bucket"
    bucket = s3_resource.Bucket(bucket_name)
    try:
        for o in bucket.objects.all():
            print(f"\t{o.key}")
    except:
        print("Error")
if __name__ == "__main__":
    list_bucket_contents(boto3.resource("s3"))

I was wondering if anyone had any insight on the following:

  1. What’s the best way to send a large amount of local data to a remote Ray Cluster?
  2. Is there an alternative to uploading this data to S3 and then setting it as the working_dir (uploading it is kind of slow)?
  3. Am I missing something with the Remote URI setup to be able to use an S3 bucket as the the working_dir?

Additional Notes:

  • I’m running Ray in Docker
  • I’m running my code in Jupyter, which connects to my remote Ray Cluster
2
  1. Yes, this is the recommended way.
  2. This is the best way.
  3. Probably the issue is that the Ray workers are not able to pick up the AWS credentials. How are you setting them up? Can you try pinging the S3 bucket from an empty Ray task first?
3

Thanks for the help!

I’ve set up the credentials both via environment variables as well as a credentials file.

I tweaked the code a bit and was able to list the contents of the bucket via Ray as well. Seems to be just the bit with the working_dir that’s failing to locate credentials.

1 Like
4

Hi @Stephanie_Wang is there anything else you can think of that might be causing Ray to be unable to find the credentials?