# ray.exceptions.WorkerCrashedError **URL:** \ **Category:** Kubernetes\ **Created:** [June 25, 2021, 11:57pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664 "2021-06-25T23:57:01Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![blu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blu/32/1099_2.png) [@blu](https://discuss.ray.io/u/blu)\ **Post date:** [June 25, 2021, 11:57pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/1 "2021-06-25T23:57:01Z")
Getting this error when running a job on ray on k8s. Job runs for ~7 hours and it’s close to completion and decides to puke out. Any ideas what’s going on? Also can anyone provide where the `python-core-worker` logs are stored? ```auto ray.exceptions.WorkerCrashedError: The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information. ``` ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [June 28, 2021, 5:54pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/2 "2021-06-28T17:54:18Z")
Hi, sorry to hear that ☹ You can check `/tmp/ray/` for the log files, in particular `/tmp/ray/session-latest/` for the logs from the most recent session. More details here: [Logging — Ray v2.0.0.dev0](https://docs.ray.io/en/master/ray-logging.html) ---
**Author:** ![blu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blu/32/1099_2.png) [@blu](https://discuss.ray.io/u/blu)\ **Post date:** [June 28, 2021, 6:37pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/3 "2021-06-28T18:37:49Z")
We are running Ray with k8s. We can’t access the logs files on the pod because it fails so we get `error: cannot exec into a container in a completed pod; current phase is Failed` We do see the logs to stdout using `kubectl logs` though, here’s the full traceback ```auto Traceback (most recent call last): File "main.py", line 193, in results = ray.get(futures) File "/usr/local/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 46, in wrapper return getattr(ray, func. __name__ )(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/ray/util/client/api.py", line 35, in get return self.worker.get(vals, timeout=timeout) File "/usr/local/lib/python3.7/site-packages/ray/util/client/worker.py", line 196, in get res = self._get(obj_ref, op_timeout) File "/usr/local/lib/python3.7/site-packages/ray/util/client/worker.py", line 219, in _get raise err ray.exceptions.WorkerCrashedError: The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information. A worker died or was killed while executing task c4280fb54ad6b274ffffffffffffffffffffffff05000000. ``` Any ideas how to troubleshoot? ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [June 28, 2021, 9:08pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/4 "2021-06-28T21:08:17Z")
I see… @Dmitri any insights here on debugging with Kubernetes? ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [June 29, 2021, 2:59am UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/5 "2021-06-29T02:59:12Z")
Logging is probably the most critical issue we haven’t addressed yet in adapting Ray for K8s. Posting a user guide for capturing logs is tracked here > > > Document how to extract logs produced by Ray processes using e.g. Loki when runn…ing K8s operator. ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [June 29, 2021, 3:06am UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/6 "2021-06-29T03:06:28Z")
@tgaddair could have some insight on debugging failures of long-running Ray jobs on K8s ---
**Author:** ![tgaddair](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tgaddair/32/1029_2.png) [@tgaddair](https://discuss.ray.io/u/tgaddair)\ **Post date:** [August 13, 2022, 6:26pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/7 "2022-08-13T18:26:15Z")
Hey @Dmitri , we use Promtail + Loki to collect all logs from Ray workers so they can be used after a worker pod crashes. ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [August 13, 2022, 11:24pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/8 "2022-08-13T23:24:40Z")
Ah, yeah, we do more or less the same to collect logs internally at Anyscale. For anyone else who lands here: The key thing to know is that Ray logs live in /tmp/ray/session\_latest/logs in the Ray container. Lofg processing tools like promtail/Loki can be used to scrape and export these logs. The docs for the upcoming Ray 2.0.0 will have some guidance on this. ---
**Author:** ![davidxia](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/davidxia/32/2951_2.png) [@davidxia](https://discuss.ray.io/u/davidxia)\ **Post date:** [August 22, 2022, 4:26pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/9 "2022-08-22T16:26:04Z")
[Some useful docs here](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/logging.html#kuberay-logging) ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [August 22, 2022, 5:09pm UTC](https://discuss.ray.io/t/ray-exceptions-workercrashederror/2664/10 "2022-08-22T17:09:49Z")
> The docs for the upcoming Ray 2.0.0 will have some guidance on this. Thank @davidxia! Precisely the docs I was referring to.