Hi Ray team,
I am trying to deploy Kuberay on our k8s cluster and integrate with flyte. We have label requirements for deployments, daemonsets, pods and statefulsets for resource tracking. I am installing the kuberay-operator, ray-cluster and kuberay-apiserver using the helm charts, I modified the templates to apply commonLabels for head nodes, workernodes and deployments. However, I am seeing this error in the kuberay-operator pod logs -
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227
2023-05-30T13:42:15.314Z DEBUG events Warning {"object": {"kind":"RayCluster","namespace":"flytesnacks-development","name":"a8mgdlwhk9x68t5rsn7f-fmxeunxq-0-raycluster-bpgwj","uid":"a858b021-d712-477f-b5de-cd9c9135d4da","apiVersion":"ray.io/v1alpha1","resourceVersion":"24421101"}, "reason": "PodReconciliationError", "message": "admission webhook \"validate.kyverno.svc-fail\" denied the request: \n\npolicy Pod/flytesnacks-development/dlwhk9x68t5rsn7f-fmxeunxq-0-raycluster-bpgwj-head-4hwnh for resource violation: \n\nrequire-labels:\n ba-label: 'validation error: The label `ba` is required. rule\n ba-label failed at path /metadata/labels/ba/'\n component-label: 'validation error: The label `/component`\n is required.
This is happening for the following types of pods as well.
Pod/flytesnacks-development/hfmtn7l7jbw2xgwf-fmxeunxq-0-raycluster-njhbc-head-mtfgn
Pod/flytesnacks-development/fx7r-fmxeunxq-0-raycluster-hhllk-worker-ray-group-xc7mx
I am not sure if I am missing adding labels anywhere in the raycluster-cluster.yaml. I think I have applied labels to RayCluster. Can someone please check and let me know if I need to add labels anywhere else in the yaml file or elsewhere to resolve this issue?
I have added this line -
{{- tpl ( .Values.commonLabels | toYaml ) . | nindent 4 }}
to all the metadata sections in raycluster-cluster.yaml and am passing in commonLabels via a custom values.yaml.
Are there any pods created by the CRDs? If so, how can we inject labels to pods created there?