Downloading working directory from private S3 storage

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi, I want to connect Ray to a private S3 storage and read working_dir files from there.

To download the working_dir I’ve set the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_ENDPOINT_URL in the env_vars field of runtime_env but get an botocore.exceptions.NoCredentialsError: Unable to locate credentials exception when Ray attempts to download the working_dir contents (remote URI s3://) . I’ve checked that the environment variables are passed correctly to the application ( I can access my S3 bucket with custom boto calls ).

I masked the variables but this is a snippet from the code:

from ray.dashboard.modules.serve.sdk import ServeSubmissionClient


client = ServeSubmissionClient("https://some-endpoint")


config = {
  "applications": [
    {
        "name": "myapp",
        "import_path": "app:myapp",
        "runtime_env": {
            "working_dir": "s3://mybucket/serve-archive.zip",
            "env_vars": {
                "AWS_ACCESS_KEY_ID": "some-key-id",
                "AWS_SECRET_ACCESS_KEY": "some-access-key",
                "AWS_ENDPOINT_URL": "https://some-endpoint",
            }
          },
        "route_prefix": "/some-route"

    }
  ]
}

client.deploy_applications(config)

How do I connect Ray properly to our S3 storage?

2

This issue seems related: Unable to locate credentials with S3 Remote URIs

3

Hi stwerner97, welcome to the Ray community! :smiley:

So there’s a few different things about authenticating with Ray, namely, Ray relies on environment variables, shared credentials files, or IAM roles for AWS authentication. It doesn’t explicitly pass credentials to boto3. Experiment Tracking — Ray 2.41.0

There might be a few different reasons why you can’t find the credentials.

  1. First can you confirm that AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_ENDPOINT_URL are correctly defined in your runtime_env’s env_vars?
  2. If they are available, can you write a small Ray task to print them out to ensure that Ray can access them properly? Maybe something like this?
import os
import ray

@ray.remote
def check_env():
    return {
        "AWS_ACCESS_KEY_ID": os.getenv("AWS_ACCESS_KEY_ID"),
        "AWS_SECRET_ACCESS_KEY": os.getenv("AWS_SECRET_ACCESS_KEY"),
        "AWS_ENDPOINT_URL": os.getenv("AWS_ENDPOINT_URL"),
    }

print(ray.get(check_env.remote()))
  1. If environment variables don’t work, have you tried adding ~/.aws/credentials or ~/.aws/config files on all nodes in your cluster?
  2. Since you’re running on S3/AWS already, think about using IAM roles with the permissions attached to the instances running Ray, which can automatically provide the required credentials.

Some more docs that might be helpful:

4

Hi @christina , thanks for the response! :slight_smile:

  1. Yes, I can confirm that. I’ve tried this with a custom script that uses boto3 to download the working directory from S3 (which I want Ray to do).
  2. Yes, this also works.
  3. I have tried this. Adding the access keys as environment variables to the cluster deployment (via secrets in Kubernetes) makes Ray set up the working dir correctly. This does not solve my use case, however, since I want other users to pass their credentials.

Thanks!

5

Hi stwerner97,
Can you elaborate a bit more on your use case, like what exactly you mean by “other users to pass their credentials”? I think that would help me debug this a lot!
Tyvm! :slight_smile:

6

Hi @christina , thanks for looking into this.

I want developers (users) to be able to deploy their own Ray serve applications on an existing Ray cluster that is deployed on Kubernetes with KubeRay. This includes, for example, to serve an object detection model with custom pre- and post-processing code. This custom serving code needs to be transferred from local storage to the remote cluster. Since uploading a local working directory does not work as it does with the Jobs API, I want to upload an archive of the user’s source code to S3 storage. Each developer has typically access to their own group’s S3 endpoint (…) which is why I’d like them to pass their credentials. If there is another good way to transfer local code to the remote cluster for serving, this would also solve my use case :slight_smile:

Best Regards