# Ray worker died from unrecoverable error but it actually keeps running
**URL:** https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857
**Category:** Dashboard, Monitoring & Debugging
**Created:** [February 28, 2024, 10:16pm UTC](https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857 "2024-02-28T22:16:16Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Jimmy\_Cao](https://discuss.ray.io/u/Jimmy_Cao)
#### Post date: [February 28, 2024, 10:16pm UTC](https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857/1 "2024-02-28T22:16:16Z")
**How severe does this issue affect your experience of using Ray?**
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
Has anyone ever encounter such a situation? Here are some relevant logs showing the supposedly dead worker (PID 253) is still running after raylet reports it is dead and has spun up a new worker to retry the task. Also, I verified the dead worker was still running externally because i can see requests coming from it in another service it was calling.
```auto
(_ray_query pid=253, ip=10.216.195.5) 2024-02-28 11:43:29,024|INFO|.... app logging
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 03ed4e0da3e7d01091cab6ee06591ae5a492dfee35000000 Worker ID: 553663a8f8d04d87d6582249470b6c814e18ee1d8b8a5ff020c69393 Node ID: cdd464a4d5397e4f15f082f65fbeb0e42f4472cd03ad71e30e9554d0 Worker IP address: 10.216.195.5 Worker port: 10002 Worker PID: 253 Worker exit type: SYSTEM_ERROR Worker exit detail: The leased worker has unrecoverable failure. Worker is requested to be destroyed when it is returned. RPC Error message: Socket closed; RPC Error details:
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffb3ec3185ceeb075b483f763835000000 Worker ID: 2d9b8f403a7be2d68d5372a5bbbd4146bf66ab440f74153c2c40cbfe Node ID: cdd464a4d5397e4f15f082f65fbeb0e42f4472cd03ad71e30e9554d0 Worker IP address: 10.216.195.5 Worker port: 10003 Worker PID: 367 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker exits unexpectedly by a signal. SystemExit is raised (sys.exit is called). Exit code: 1. The process receives a SIGTERM.
(_ray_query pid=963, ip=10.216.195.5) 2024-02-28 11:44:05,771|INFO|.... app logging
(_ray_query pid=963, ip=10.216.195.5) 2024-02-28 11:44:06,108|INFO|.... app logging
(_ray_query pid=253, ip=10.216.195.5) 2024-02-28 11:44:13,159|INFO|.... app logging
```
Also, this worker PID 253 kept running for about an hour after I stopped all jobs on the cluster. Eventually, the worker node died on its own.
Update: This seems to consistently happen, the raylet will report the worker is dead after one long running task executes for about 1 hour 10mins. Guess we’re hitting some timeout?
---
### Author:  [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)
#### Post date: [February 28, 2024, 10:48pm UTC](https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857/2 "2024-02-28T22:48:38Z")
@yic could you take a look?
---
### Author:  [@yic](https://discuss.ray.io/u/yic)
#### Post date: [February 29, 2024, 3:32am UTC](https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857/3 "2024-02-29T03:32:16Z")
Hi @Jimmy_Cao it seems like a bug. Do you minding sharing your script for this?
If not, can you share the raylet logs and the the dead worker logs ? They should be in `/tmp/ray/session_latest/` on your local node.
---
### Author:  [@Jimmy\_Cao](https://discuss.ray.io/u/Jimmy_Cao)
#### Post date: [February 29, 2024, 6:58pm UTC](https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857/4 "2024-02-29T18:58:20Z")
Thanks! I can’t share the exact script but let me see if i can share a minimal repro. I don’t have the logs anymore as i had to restart my cluster
---
### Author:  [@Jimmy\_Cao](https://discuss.ray.io/u/Jimmy_Cao)
#### Post date: [March 1, 2024, 3:18am UTC](https://discuss.ray.io/t/ray-worker-died-from-unrecoverable-error-but-it-actually-keeps-running/13857/5 "2024-03-01T03:18:22Z")
This reproduces the bug, I’m not sure if the Actor is needed:
```auto
import time
from collections import defaultdict
from typing import DefaultDict
import ray
@ray.remote(num_cpus=1)
class StatsActor:
def __init__ (self) -> None:
self._stats: DefaultDict[str, int] = defaultdict(int)
def increment(self, stat_key: str) -> None:
self._stats[stat_key] += 1
@ray.remote
def long_running_task():
actor_ref = StatsActor.remote()
for idx in range(2*60*60): # run for ~2hours, should fail around 1h 10mins
if idx % 100 == 0:
print("im alive!")
for _ in range(30):
# do some work at 30qps and incr a counter
time.sleep(1/30)
actor_ref.increment.remote("somekey")
ray.init()
ray.get(long_running_task.remote(), timeout=3*60*60) # timeout longer than task run time
```