Docker volume not being mounted

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I have a local cluster that I set up through the cluster launcher with docker enabled. However, the volumes that I specify to be mounted into the container dont get mounted.

This is my cluster config yaml:

# A unique identifier for the head node and workers of this cluster.
cluster_name: default

# Running Ray in Docker images is optional (this docker section can be commented out).
# This executes all commands on all nodes in the docker container,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled. Assumes Docker is installed.
docker:
    image: "rayproject/ray-ml:latest-gpu" # You can change this to latest-cpu if you don't need GPU support and want a faster startup
    # image: rayproject/ray:latest-gpu   # use this one if you don't need ML dependencies, it's faster to pull
    container_name: "ray_container"
    # If true, pulls latest version of image. Otherwise, `docker run` will only pull the image
    # if no cached version is present.
    pull_before_run: True
    run_options:   # Extra options to pass into "docker run"
        - --ulimit nofile=65536:65536
    head_run_options:   # Extra options to pass into "docker run" for the head node
        # mout ~/data to ~/data in the container
        - "-v /home/myuser/data:/home/ray/data:z" # doesn't work without  :z either

# etc....

auth:
    ssh_user: myuser

# etc...

Everything else works fine (conda envs and so on).

Does someone know what is wrong with my docker config? Or where can I see logs/errors with regards to the docker command.

(I ssh into the head node with ray attach to check if the folder is there or not. ATM I only have a head node)

2

Ok it works now. I changed the name of one of my mile mounts from ray to raycfg (i was thinking maybe the ‘ray’ name was causing issues) which seemed to trigger the docker container to be recreated and now the data folder is there. Don’t know if the former or the latter resolved the issue.

1 Like