Hi!
I tried creating a simple cluster with the following minimal configuration:
cluster_name: minimal
provider:
type: aws
region: ap-south-1
This is resulting in the following error message:
2022-02-02 10:34:46,317 INFO util.py:282 -- setting max workers for head node type to 0
2022-02-02 10:34:46,317 INFO util.py:286 -- setting max workers for ray.worker.default to 2
Checking AWS environment settings
Failed to autodetect node resources: An error occurred (UnauthorizedOperation) when calling the DescribeInstanceTypes operation: You are not authorized to perform this operation.. You can see full stack trace with higher verbosity.
Failed to fetch IAM instance profile data for ray-autoscaler-v1 from AWS.
Error code: AccessDenied
!!! Boto3 error:
An error occurred (AccessDenied) when calling the GetInstanceProfile operation: User: arn:aws:sts::847438129436:assumed-role/data-science-emr-s3-access-rw/SageMaker is not authorized to perform: iam:GetInstanceProfile on resource: instance profile ray-autoscaler-v1
Could you guys please help me with this?
Roles that are required to be manually setup
Permissions that are required to be manually setup
Thanks,
1 Like
@ijrsvt , can you please help?
CC @sasha-s
I’m having the same issue. Can’t find anything in ray docs.
Sasha
April 14, 2022, 6:28pm
4
Please have a look at the docshttps://docs.ray.io/en/latest/cluster/cloud.html
opened 05:57AM - 07 Jul 20 UTC
fix-docs
infra
This (non) issue takes a brief look at how we can minimize the permissions grant… ed to the Ray Cluster Launcher when using it with AWS.
The cluster launcher works by launching a single head node and using that node to launch the cluster’s worker nodes. If you’re using the launcher with AWS for the first time, an Instance Profile is auto-created and a role with full EC2 and S3 permissions is attached to it; this role also has the `sts:AssumeRole` permission.
This works seamlessly for basic use-cases, but if you need to grant AWS permissions to the worker nodes – to allow them to access S3, for example – you’re going to need to make a few changes. While we’re doing that, let’s also trim down the EC2 and S3 permissions granted to the head node.
### Example Use Case
Let’s say we need a setup that has the following properties:
* The Ray Cluster Launcher is allowed to launch instances only in the us-west-1 region.
* The head and the worker nodes will have access to the `ray-data` S3 bucket.
### Breakdown
* The console you’re using to launch the cluster (launchpad) needs permissions to launch instances in the us-west-1 region. It also needs to assign an IAM role to the head node.
* The head node needs similar permissions since it has to launch worker nodes in the same region and pass an IAM role to each one. It will also need to access the `ray-data` S3 bucket
* The worker nodes will only need permissions to access the `ray-data` bucket.
### Steps
#### 1. Create an IAM role to assign to the head node
Role name: `ray-head-v1`
If you create this role for EC2 on the AWS console, an instance profile will be automatically created.
If you create this role using the AWS CLI, then create an instance profile of the same name and assign the role to it as below.
```bash
aws iam create-instance-profile --instance-profile-name ray-head-v1
aws iam add-role-to-instance-profile --instance-profile-name ray-head-v1 --role-name ray-head-v1
```
The AWS console page for this role will also list the ARN for the instance profile. Or to access it with the CLI:
```bash
aws iam list-instance-profiles | grep ray-head-v1
```
#### 2. Create an IAM role to assign to the worker node
Role name: `ray-worker-v1`
Follow the same procedure as the previous step.
#### 3. Create an IAM policy that will allow EC2 instance launches
Policy name: `ray-ec2-launcher`
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:us-west-1::image/ami-*"
},
{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:us-west-1:<aws-account-number>:instance/*",
"arn:aws:ec2:us-west-1:<aws-account-number>:network-interface/*",
"arn:aws:ec2:us-west-1:<aws-account-number>:subnet/*",
"arn:aws:ec2:us-west-1:<aws-account-number>:key-pair/*",
"arn:aws:ec2:us-west-1:<aws-account-number>:volume/*",
"arn:aws:ec2:us-west-1:<aws-account-number>:security-group/*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:TerminateInstances",
"ec2:DeleteTags",
"ec2:StartInstances",
"ec2:CreateTags",
"ec2:StopInstances"
],
"Resource": "arn:aws:ec2:us-west-1:<aws-account-number>:instance/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:Describe*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::<aws-account-number>:instance-profile/ray-head-v1",
"arn:aws:iam::<aws-account-number>:instance-profile/ray-worker-v1"
]
}
]
}
```
#### 4. Create a policy to access the S3 bucket
Policy name: `ray-s3-access`
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::ray-data/*",
"arn:aws:s3:::ray-data",
]
}
]
}
```
#### 5. Assign both of the above policies to the `ray-head-v1` role
You can do this either through the AWS console interactively or using the CLI with:
```bash
aws iam attach-role-policy --policy-arn arn:aws:iam::<aws-account-number>:policy/ray-ec2-launcher --role-name ray-head-v1
aws iam attach-role-policy --policy-arn arn:aws:iam::<aws-account-number>:policy/ray-s3-access --role-name ray-head-v1
```
#### 6. Assign the S3 access policy to the `ray-worker-v1` role
#### 7. Assign the `ray-ec2-launcher policy` to a launchpad role/user
This can optionally be done to limit the permissions assigned to the role/user that will be operating the Ray cluster launcher. For example, if you’re an AWS administrator and need to allow one of your users to (only) launch Ray clusters.
#### 8. Edit your cluster config YAML file
Under `head_node:`, add:
```yaml
IamInstanceProfile:
Arn: arn:aws:iam::<aws-account-number>:instance-profile/ray-head-v1
```
Under `worker_nodes:`, add:
```yaml
IamInstanceProfile:
Arn: arn:aws:iam::<aws-account-number>:instance-profile/ray-worker-v1
```
### Summary
While the `ray-ec2-launcher` policy has reduced permissions compared to the original, it’s still possible to whittle this down further by specifying the AMIs, subnets, key-pairs, etc that the cluster launcher is allowed to access, as opposed to using a wildcard.
has an example of an AWS setup with S3 access.