Unable to attach an EFS volume to autoscaler ray cluster on AWS. b' mount.nfs4: Operation not permitted'

1

I have been tearing my hear out about setting up an EFS volumn to my ray cluster.

I keep getting

mount.nfs4: Operation not permitted

I have the correct permissions in the role, I have the correct policies for the network (they are in the same security group) and it is able to find the EFS volume, but the last step is just not getting through.

I have installed the efs-utils and botocore as instructed, but whatever I try, NFS refused to mount the EFS vlue.

setup_commands:
     - sudo mkdir /efs
     - sudo mount -t efs fs-00f2a6a0aacee8c71  /efs
     - sudo chmod -R 777 /efs

i tried setting it up in home directory as well to no avail. I do notifce that nfs is not running as as ervice, but that is probably intended.

NE1 have any idea?

2

SOLVED: The docker needs to be run in PRIVILIGED mode.
so needed to add “–priviliged” to the autoscaler.yaml file. This would be useful to see mentioned in the docs…just sayin’ :slight_smile:

docker:
    image: "rayproject/ray-ml:latest-gpu" # You can change this to latest-cpu if you don't need GPU support and want a faster startup
    # image: rayproject/ray:latest-cpu   # use this one if you don't need ML dependencies, it's faster to pull
    container_name: "ray_container"
    # If true, pulls latest version of image. Otherwise, `docker run` will only pull the image
    # if no cached version is present.
    pull_before_run: True
    run_options:   # Extra options to pass into "docker run"
        - --ulimit nofile=65536:65536
        - --shm-size=10g
        - --privileged