Template change for workers should trigger a pod recreate

Ray Clusters Kubernetes
1

Hi,

I’m using a Ray Cluster on Kubernetes with a custom image. I.e.:

  headGroupSpec:
    # ...
    #pod template
    template:
      spec:
        containers:
          - name: ray-head
            image: img:tag

When I change the tag due to an update, the operator doesn’t roll out the update, i.e. it doesn’t delete the ray-head pod and recreate it from the updated template.
If I manually delete the pod with kubectl delete pod raycluster-head-**** then it triggers the creation from the up to date template.

Is this intentional?
I assume the rollout needs to be done carefully, just pulling the plug on the ray-head pod is not ideal, because it crashes any running Ray processes (I think?).

So, what’s the recommended workflow for updating templates automatically?

2

Delete and recreate.
The problem is known and tracked here: [Feature] rolling upgrade design and implementation for Kuberay · Issue #527 · ray-project/kuberay · GitHub