K8s cluster does not set serviceaccounts correctly

I’m having issues setting permissions using the kubernetes operator.

I can’t pull containers from google cloud registry on the head or worker nodes, though the operator node is able to.

I have the ray_operator_serviceaccount configured to bind to a GCP service account using the instructions in Using Workload Identity  |  Kubernetes Engine Documentation.

The service account works if I manually set the ServiceAccountName field in the pod spec on a normal k8s workload, just not through ray.

CC @Dmitri , any suggestions?

Sounds like in your case the head and workers also need access to a k8s service account bound to a GCP service account.

Two options:
(1) [recommended]
Create a K8s service account for this purpose,
and specify this service account in the pod specs in your RayCluster custom resource

(2) [not recommended but will work faster for a quick test]
Specify your already existing ray_operator_serviceaccount in the pod specs for the head and worker nodes.

(2) Is not ideal from a security perspective – the operator has a bunch of permissions (e.g. pod creation) that the pods in your ray cluster don’t need.

But bottom line is that whichever pod is pulling containers from your gcr registry needs to use a service account configured for that.

(Or is that already the case in your setup?)

Please let me know if you have further questions.