GCP Autoscaler: Solve Artifacts "Permission denied" error

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • None: Just giving advice to other users

If you use the autoscaler for your Ray cluster on GCP and want to use an image you pushed to the GCP Artifacts Registry and then you encounter the following error:

Error response from daemon: Head "https://<LOCATION>-docker.pkg.dev/v2/<PPROJECT-ID>/<REGISTRY>/<IMAGE>/manifests/<TAG>": denied: 
Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/>PROJECT-ID>/locations/<LOCATION>/repositories/<REGISTRY>" (or it may not exist)

where is the location of your registry, check the following:

  1. The Ray Autoscaler Service Account is given the Artifacts Registry Reader role binding (or a role binding with even more rights).
  2. In the initialization commands attribute of your autoscaler yaml file you added the following command: gcloud auth configure-docker <LOCATION>-docker.pkg.dev --quiet

Has cost me some hours to figure this out, so I thought this helps others to spare their time.

1 Like
2

Thanks for sharing!

If more users run into this, perhaps it would be worth a tip on the docs.

1 Like