# Head pod does not restart after deleting/draining

**URL:** <https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187>\
**Category:** Kubernetes\
**Created:** [May 19, 2022, 9:18am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187 "2022-05-19T09:18:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrew\_Li](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andrew_li/32/2425_2.png) [@Andrew\_Li](https://discuss.ray.io/u/Andrew_Li)\
**Post date:** [May 19, 2022, 9:18am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/1 "2022-05-19T09:18:01Z")

</div>

Hi there,

I am deploying Ray on K8s using [Helm Chart](https://github.com/ray-project/ray/tree/master/deploy/charts/ray), the environment and the `values.yaml` shows below:

- ray == 1.12.0
- runing ray on K8s(GKE)
  - 1.21.10-gke.2000

```nohighlight
# `values.yaml`
image: rayproject/ray:1.12.0-py38
podTypes:
  rayHeadType
    CPU: 4
    memory: 30Gi
    GPU: 0
    rayResources: { "CPU": 0 }
  rayWorkerType:
    minWorkers: 0
    maxWorkers: 6
    memory: 30Gi
    CPU: 3
    GPU: 0

```

For any reason, when the head pod was deleted or the node was drained where the head pod lived, the head pod will never be created again.

Now, my workaround is slightly to modify the spec of `rayHeadType` forcing to restart the Ray cluster([Ray Operator Advanced Configuration — Ray 1.12.1](https://docs.ray.io/en/latest/cluster/kubernetes-advanced.html#restart-behavior)). For instance, I changed the CPU of `rayHeadType` to `3` and changed it back to `4` again.

Any tips for this situation?

---

<div class="post-metadata">

**Author:** ![Andrew\_Li](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andrew_li/32/2425_2.png) [@Andrew\_Li](https://discuss.ray.io/u/Andrew_Li)\
**Post date:** [August 5, 2022, 4:21am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/2 "2022-08-05T04:21:17Z")

</div>

Hi there, any suggestion?

---

<div class="post-metadata">

**Author:** ![cade](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cade/32/3837_2.png) [@cade](https://discuss.ray.io/u/cade)\
**Post date:** [August 9, 2022, 1:23am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/3 "2022-08-09T01:23:33Z")

</div>

cc @Dmitri who knows a lot about k8s and Ray

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [August 9, 2022, 2:08am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/4 "2022-08-09T02:08:50Z")

</div>

I’d recommend checking out KubeRay [GitHub - ray-project/kuberay: A toolkit to run Ray applications on Kubernetes](https://github.com/ray-project/kuberay) to deploy Ray on Kubernetes.  
Detailed KubeRay documentation and guides are in the works.

---

<div class="post-metadata">

**Author:** ![Andrew\_Li](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andrew_li/32/2425_2.png) [@Andrew\_Li](https://discuss.ray.io/u/Andrew_Li)\
**Post date:** [August 9, 2022, 4:04am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/5 "2022-08-09T04:04:40Z")

</div>

Thanks for working on KubeRay. I did check out this repo but I couldn’t figure out what’s different from the current process of deploying on K8s([Deploying on Kubernetes — Ray 1.13.0](https://docs.ray.io/en/latest/cluster/kubernetes.html#installing-the-ray-operator-with-helm)). Hope the detailed documentation will point it out.

---

<div class="post-metadata">

**Author:** ![Andrew\_Li](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andrew_li/32/2425_2.png) [@Andrew\_Li](https://discuss.ray.io/u/Andrew_Li)\
**Post date:** [August 9, 2022, 4:07am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/6 "2022-08-09T04:07:52Z")

</div>

Since both [Installing the Ray Operator with Helm](https://docs.ray.io/en/latest/cluster/kubernetes.html#installing-the-ray-operator-with-helm) and [KubeRay](https://github.com/ray-project/kuberay) are using Helm to manage it. Is it going to have an official Helm repo instead of cloning the whole GitHub repo?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [August 9, 2022, 4:51am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/7 "2022-08-09T04:51:21Z")

</div>

We will eventually host Helm charts for KubeRay at a public Helm repository.

The primary difference between KubeRay and the operator currently documented in the Ray repo is, frankly, that KubeRay is more stable – e.g. you should not have issues with a head pod that fails to restart.

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [August 9, 2022, 5:48am UTC](https://discuss.ray.io/t/head-pod-does-not-restart-after-deleting-draining/6187/8 "2022-08-09T05:48:26Z")

</div>

Though returning to the original question, head restarts with the older operator are tested, though I’ve observed that the restart might take a minute or so.  
If you could post operator logs (kubectl logs ray-opertor) from when the head fails to restart, that would be helpful.