Template change for workers should trigger a pod recreate


I’m using a Ray Cluster on Kubernetes with a custom image. I.e.:

    # ...
    #pod template
          - name: ray-head
            image: img:tag

When I change the tag due to an update, the operator doesn’t roll out the update, i.e. it doesn’t delete the ray-head pod and recreate it from the updated template.
If I manually delete the pod with kubectl delete pod raycluster-head-**** then it triggers the creation from the up to date template.

Is this intentional?
I assume the rollout needs to be done carefully, just pulling the plug on the ray-head pod is not ideal, because it crashes any running Ray processes (I think?).

So, what’s the recommended workflow for updating templates automatically?

Delete and recreate.
The problem is known and tracked here: [Feature] rolling upgrade design and implementation for Kuberay · Issue #527 · ray-project/kuberay · GitHub