# Killing driver does not kill tasks in Ray on minikube

**URL:** <https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790>\
**Category:** Kubernetes\
**Created:** [April 19, 2021, 10:05pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790 "2021-04-19T22:05:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [April 19, 2021, 10:05pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/1 "2021-04-19T22:05:43Z")

</div>

Hello,

I use sample program available from one of the ray tutorial and run below code snippet on the ray head node CLI:

@ray.remote  
def f():  
time.sleep(2)

#call ray init

result = ray.get(f.remote for \_ in range 1500)

on head node CLI when I execute the above command a demand of 1500 CPUs is created. In my minikube env I do not have 1500 CPUs, so I think tasks are executed in ray defined order.

When I exit the CLI command by ctrl+Z, while the 1500 tasks are completing I was expecting that the remaining tasks should have been terminated but that is not the case. The tasks keep on executing even when I have killed the CLI command.

When I resubmit 1500 tasks again from CLI, I am not sure in what order the tasks are executed, will ray finish my killed CLI task first before executing the new tasks?

Can you please comment if this is expected?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [April 20, 2021, 3:28pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/2 "2021-04-20T15:28:45Z")

</div>

Sounds like a bug  
@sangcho exiting the driver should cancel pending tasks, right?

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [April 20, 2021, 6:22pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/3 "2021-04-20T18:22:43Z")

</div>

Did you call ray.init() or ray.init(address=‘auto’)?

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [April 20, 2021, 6:26pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/4 "2021-04-20T18:26:06Z")

</div>

I think I called ray.init(address=‘auto’)

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [April 20, 2021, 6:34pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/5 "2021-04-20T18:34:37Z")

</div>

> The tasks keep on executing even when I have killed the CLI command.

Does that mean tasks kept executing until all of them are finished?

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [April 20, 2021, 6:35pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/6 "2021-04-20T18:35:18Z")

</div>

My expectation is that when the driver exits, pending tasks shouldn’t be executed. cc @Alex Can you follow up if my understanding is correct?

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [April 20, 2021, 6:35pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/7 "2021-04-20T18:35:37Z")

</div>

Yes it kept executing even when the driver was killed

---

<div class="post-metadata">

**Author:** ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)\
**Post date:** [April 20, 2021, 7:04pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/8 "2021-04-20T19:04:04Z")

</div>

@asm582 I think the issue here is that you are intending to kill the program, but you’re actually stopping it instead.

In general, ctrl+z sends a SIGSTOP to the program, which only pauses it, so the tasks remain because you could always unpause the program by running `kill -SIGCONT <pid>`.

I think what you’re looking for is ctrl+c which will actually kill the process and trigger the cleanup (SIGTERM).

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [April 22, 2021, 10:35pm UTC](https://discuss.ray.io/t/killing-driver-does-not-kill-tasks-in-ray-on-minikube/1790/10 "2021-04-22T22:35:25Z")

</div>

Thank you, I now confirm with SIGTERM the tasks are killed
