Labelling all resources created by KubeRay on k8s

Hi Ray team,
I am trying to deploy Kuberay on our k8s cluster and integrate with flyte. We have label requirements for deployments, daemonsets, pods and statefulsets for resource tracking. I am installing the kuberay-operator, ray-cluster and kuberay-apiserver using the helm charts, I modified the templates to apply commonLabels for head nodes, workernodes and deployments. However, I am seeing this error in the kuberay-operator pod logs -*Controller).processNextWorkItem
2023-05-30T13:42:15.314Z	DEBUG	events	Warning	{"object": {"kind":"RayCluster","namespace":"flytesnacks-development","name":"a8mgdlwhk9x68t5rsn7f-fmxeunxq-0-raycluster-bpgwj","uid":"a858b021-d712-477f-b5de-cd9c9135d4da","apiVersion":"","resourceVersion":"24421101"}, "reason": "PodReconciliationError", "message": "admission webhook \"validate.kyverno.svc-fail\" denied the request: \n\npolicy Pod/flytesnacks-development/dlwhk9x68t5rsn7f-fmxeunxq-0-raycluster-bpgwj-head-4hwnh for resource violation: \n\nrequire-labels:\n  ba-label: 'validation error: The label `ba` is required.  rule\n    ba-label failed at path /metadata/labels/ba/'\n  component-label: 'validation error: The label `/component`\n    is required.

This is happening for the following types of pods as well.


I am not sure if I am missing adding labels anywhere in the raycluster-cluster.yaml. I think I have applied labels to RayCluster. Can someone please check and let me know if I need to add labels anywhere else in the yaml file or elsewhere to resolve this issue?

I have added this line -

{{- tpl ( .Values.commonLabels | toYaml ) . | nindent 4 }}

to all the metadata sections in raycluster-cluster.yaml and am passing in commonLabels via a custom values.yaml.

Are there any pods created by the CRDs? If so, how can we inject labels to pods created there?