I’m using the helm chart to deploy ray cluster to GKE. I build my own ray image base on ray’s image rayproject/ray:2.5.0-py310 to install my custom dependencies etc.
I’m deploying the ray cluster helm chart in our CICD pipeline which means that it will take latest image tag from the current branch.
The raycluster is updated, I can confirm that with kubectl describe raycluster, I can see that both head and worker group has the same new image.
However only the worker group’s image is updated, of course the worker node is restarted which is expected. The head node is not updated at all, still has the old image.
Is this expected? Do we always have to re-install the helm chart?
Is this the expected behavior? I guess killing the head node will bring some risks, how to safely upgrade the head node image then?