# Ray worker dies with SYSTEM\_ERROR\_EXIT **URL:** \ **Category:** Ray Core\ **Created:** [June 27, 2022, 5:16pm UTC](https://discuss.ray.io/t/ray-worker-dies-with-system-error-exit/6642 "2022-06-27T17:16:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![elo](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elo](https://discuss.ray.io/u/elo)\ **Post date:** [June 27, 2022, 5:16pm UTC](https://discuss.ray.io/t/ray-worker-dies-with-system-error-exit/6642/1 "2022-06-27T17:16:32Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. Hi I am running Ray on a Slurm cluster where I start a lot of tasks which in turn I cancel quite often. A task takes on average 20 seconds to finish, where I might cancel a task only a few seconds after I started it. I catch the ray.cancel within the task as a KeyboardInterrupt where I then do some cleaning up. My issue is sometimes when I cancel a task, the cancelation does not seem to succeed. In particular, the KeyboardInterrupt is not received by the task. It seems like the worker the task is running on just dies silently. I can find the following output in the GCS log output: ```auto [2022-06-27 18:19:50,327 W 139409 139409] (gcs_server) gcs_worker_manager.cc:43: Reporting worker exit, worker id = 9e39c872f30f95b8c1083d1f78f21ef366bb09034f1369544492c934, node id = 34453e0c12fe101f86f3c7cf33a6f02d83182c6da8c8ea2be7d59a9f, address = 10.10.70.122, exit_type = SYSTEM_ERROR_EXIT0. Unintentional worker failures have been reported. If there are lots of this logs, that might indicate there are unexpected failures in the cluster. [2022-06-27 18:19:50,327 W 139409 139409] (gcs_server) gcs_actor_manager.cc:848: Worker 9e39c872f30f95b8c1083d1f78f21ef366bb09034f1369544492c934 on node 34453e0c12fe101f86f3c7cf33a6f02d83182c6da8c8ea2be7d59a9f exits, type=SYSTEM_ERROR_EXIT, has creation_task_exception = 0 ``` In addition, the worker logs show no more activity after the cancel. Is there a way to at least catch this inactive workers while running? It would be fine if I had to restart the task for this… Thank you! ---
**Author:** ![Chen\_Shen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chen_shen/32/1486_2.png) [@Chen\_Shen](https://discuss.ray.io/u/Chen_Shen)\ **Post date:** [June 28, 2022, 7:10am UTC](https://discuss.ray.io/t/ray-worker-dies-with-system-error-exit/6642/2 "2022-06-28T07:10:25Z")
hi @elo, depending on the state of the task there might be race conditions between the cancel signal and worker termination signal; Also I’m not sure I fully understand your way of canceling tasks > I catch the ray.cancel within the task as a KeyboardInterrupt where I then do some cleaning up. Do you have a simple repro script for the issues you are encountering? ---
**Author:** ![elo](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@elo](https://discuss.ray.io/u/elo)\ **Post date:** [June 28, 2022, 7:31am UTC](https://discuss.ray.io/t/ray-worker-dies-with-system-error-exit/6642/3 "2022-06-28T07:31:08Z")
Hi, thank you for your answer. I am not sure about the race condition. I send the cancel signal way before the task would be done. What may happen is: in the task I cancel, I call ray.remote on some actor to send data over. May receiving the cancel signal while calling ray.remote lead to this issue? The way I am canceling tasks is: I have a monitor function running as ray actor that periodically checks some conditions. In case these are fulfilled, it sends a cancel signal thorough ray.cancel to the task. In the task itself, I catch the signal as KeyboardInterrupt, do some processing and then return. The return is important since in a main loop I am using wait to get the tasks results to decided which tasks to run next. I hope that makes sense. Unfortunately, I do not have a simple script since, as described above it is rather complex. I will try to get one but I am not sure that will work since on the cluster the error only occurs very seldom. Is there any way I could check for the dead worker within the skript to then just restart the task? Thanky you ---
**Author:** ![Stephanie\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/stephanie_wang/32/18_2.png) [@Stephanie\_Wang](https://discuss.ray.io/u/Stephanie_Wang)\ **Post date:** [June 28, 2022, 5:52pm UTC](https://discuss.ray.io/t/ray-worker-dies-with-system-error-exit/6642/4 "2022-06-28T17:52:20Z")
If you are using non-actor tasks, you can rely on Ray’s internal task retries. It should actually restart the task automatically if the original worker dies. You can find more information about this feature [here](https://docs.ray.io/en/latest/ray-core/tasks/fault-tolerance.html) in the docs.