# Head and worked node dies after few seconds

**URL:** <https://discuss.ray.io/t/head-and-worked-node-dies-after-few-seconds/9878>\
**Category:** Kubernetes\
**Created:** [March 24, 2023, 5:33am UTC](https://discuss.ray.io/t/head-and-worked-node-dies-after-few-seconds/9878 "2023-03-24T05:33:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![VD23](https://avatars.discourse-cdn.com/v4/letter/v/e8c25b/32.png) [@VD23](https://discuss.ray.io/u/VD23)\
**Post date:** [March 24, 2023, 5:33am UTC](https://discuss.ray.io/t/head-and-worked-node-dies-after-few-seconds/9878/1 "2023-03-24T05:33:29Z")

</div>

Hi! I am referring the documentation [Getting Started — Ray 3.0.0.dev0](https://docs.ray.io/en/master/cluster/kubernetes/getting-started.html) for setting up ray cluster on my local machine using docker.

I followed through all the steps and I could see both head and worker pods running. However on the dashboard both the nodes turn DEAD immediately after few seconds.

**kubectl get pods**

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/3/3afdcdd2b4f7efac6167b4a0cc00991c739e9bdb.png)

**Dashboard**

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/0/0ddb236074fab6a2e1a4950a9645f2f5769679b4.png)

I’m using the sample Ray Cluster CR recommended in the documentation without any changes.

Below are the logs files picked up from the head node /tmp/ray/session\_latest/logs directory

**raylet.err**  
[2023-03-23 22:01:27,704 E 229 336] (raylet) agent\_manager.cc:135: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. Agent can fail when

- The version of `grpcio` doesn’t follow Ray’s requirement. Agent can segfault with the incorrect `grpcio` version. Check the grpcio version `pip freeze | grep grpcio`.
- The agent failed to start because of unexpected error or port conflict. Read the log `cat /tmp/ray/session_latest/dashboard_agent.log`. You can find the log file structure here [Logging — Ray 3.0.0.dev0](https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure).
- The agent is killed by the OS (e.g., out of memory).

**dashboard\_agent.log**  
2023-03-23 22:01:25,093 INFO agent.py:105 – Parent pid is 229  
2023-03-23 22:01:25,147 INFO agent.py:131 – Dashboard agent grpc address: 0.0.0.0:57134  
2023-03-23 22:01:25,156 WARNING agent.py:197 – Raylet is considered dead 1 X. If it reaches to 5, the agent will kill itself. Parent: None, parent\_gone: True, init\_assigned\_for\_parent: False, parent\_changed: False.  
2023-03-23 22:01:25,161 INFO utils.py:112 – Get all modules by type: DashboardAgentModule  
2023-03-23 22:01:26,209 INFO utils.py:145 – Available modules: [\<class ‘ray.dashboard.modules.event.event\_agent.EventAgent’\>, \<class ‘ray.dashboard.modules.healthz.healthz\_agent.HealthzAgent’\>, \<class ‘ray.dashboard.modules.job.job\_agent.JobAgent’\>, \<class ‘ray.dashboard.modules.log.log\_agent.LogAgent’\>, \<class ‘ray.dashboard.modules.log.log\_agent.LogAgentV1Grpc’\>, \<class ‘ray.dashboard.modules.reporter.reporter\_agent.ReporterAgent’\>, \<class ‘ray.dashboard.modules.runtime\_env.runtime\_env\_agent.RuntimeEnvAgent’\>, \<class ‘ray.dashboard.modules.serve.serve\_agent.ServeAgent’\>]  
2023-03-23 22:01:26,210 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.event.event\_agent.EventAgent’\>  
2023-03-23 22:01:26,211 INFO event\_agent.py:38 – Event agent cache buffer size: 10240  
2023-03-23 22:01:26,212 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.healthz.healthz\_agent.HealthzAgent’\>  
2023-03-23 22:01:26,212 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.job.job\_agent.JobAgent’\>  
2023-03-23 22:01:26,212 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.log.log\_agent.LogAgent’\>  
2023-03-23 22:01:26,212 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.log.log\_agent.LogAgentV1Grpc’\>  
2023-03-23 22:01:26,213 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.reporter.reporter\_agent.ReporterAgent’\>  
2023-03-23 22:01:26,227 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.runtime\_env.runtime\_env\_agent.RuntimeEnvAgent’\>  
2023-03-23 22:01:26,229 INFO agent.py:161 – Loading DashboardAgentModule: \<class ‘ray.dashboard.modules.serve.serve\_agent.ServeAgent’\>  
2023-03-23 22:01:26,229 INFO agent.py:165 – Loaded 8 modules.  
2023-03-23 22:01:26,245 WARNING agent.py:197 – Raylet is considered dead 2 X. If it reaches to 5, the agent will kill itself. Parent: None, parent\_gone: True, init\_assigned\_for\_parent: False, parent\_changed: False.  
2023-03-23 22:01:26,246 INFO http\_server\_agent.py:75 – Dashboard agent http address: 0.0.0.0:52365  
2023-03-23 22:01:26,246 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<PlainResource /api/local\_raylet\_healthz\> → \<function HealthzAgent.health\_check at 0x408cc92830\>  
2023-03-23 22:01:26,247 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<PlainResource /api/local\_raylet\_healthz\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,247 INFO http\_server\_agent.py:81 – \<ResourceRoute [POST] \<PlainResource /api/job\_agent/jobs/\> → \<function JobAgent.submit\_job at 0x408ccd2200\>  
2023-03-23 22:01:26,247 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<PlainResource /api/job\_agent/jobs/\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,247 INFO http\_server\_agent.py:81 – \<ResourceRoute [POST] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}/stop\> → \<function JobAgent.stop\_job at 0x408ccd2b00\>  
2023-03-23 22:01:26,248 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}/stop\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,248 INFO http\_server\_agent.py:81 – \<ResourceRoute [DELETE] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}\> → \<function JobAgent.delete\_job at 0x408ccda3b0\>  
2023-03-23 22:01:26,248 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,248 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}/logs\> → \<function JobAgent.get\_job\_logs at 0x408ccda560\>  
2023-03-23 22:01:26,249 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}/logs\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,249 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}/logs/tail\> → \<function JobAgent.tail\_job\_logs at 0x408ccda710\>  
2023-03-23 22:01:26,249 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<DynamicResource /api/job\_agent/jobs/{job\_or\_submission\_id}/logs/tail\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,249 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<PlainResource /api/ray/version\> → \<function ServeAgent.get\_version at 0x408d3f0170\>  
2023-03-23 22:01:26,250 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<PlainResource /api/ray/version\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,250 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<PlainResource /api/serve/deployments/\> → \<function ServeAgent.get\_all\_deployments at 0x408d3f0200\>  
2023-03-23 22:01:26,250 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<PlainResource /api/serve/deployments/\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,250 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<PlainResource /api/serve/deployments/status\> → \<function ServeAgent.get\_all\_deployment\_statuses at 0x408d3f03b0\>  
2023-03-23 22:01:26,251 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<PlainResource /api/serve/deployments/status\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,251 INFO http\_server\_agent.py:81 – \<ResourceRoute [DELETE] \<PlainResource /api/serve/deployments/\> → \<function ServeAgent.delete\_serve\_application at 0x408d3f0560\>  
2023-03-23 22:01:26,251 INFO http\_server\_agent.py:81 – \<ResourceRoute [PUT] \<PlainResource /api/serve/deployments/\> → \<function ServeAgent.put\_all\_deployments at 0x408d3f0710\>  
2023-03-23 22:01:26,251 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<PlainResource /api/serve/deployments/\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,252 INFO http\_server\_agent.py:81 – \<ResourceRoute [GET] \<StaticResource /logs → PosixPath(‘/tmp/ray/session\_2023-03-23\_22-01-12\_707254\_14/logs’)\> → \<bound method StaticResource.\_handle of \<StaticResource /logs → PosixPath(‘/tmp/ray/session\_2023-03-23\_22-01-12\_707254\_14/logs’)\>\>  
2023-03-23 22:01:26,252 INFO http\_server\_agent.py:81 – \<ResourceRoute [OPTIONS] \<StaticResource /logs → PosixPath(‘/tmp/ray/session\_2023-03-23\_22-01-12\_707254\_14/logs’)\> → \<bound method \_PreflightHandler.\_preflight\_handler of \<aiohttp\_cors.cors\_config.\_CorsConfigImpl object at 0x408d458d90\>\>  
2023-03-23 22:01:26,252 INFO http\_server\_agent.py:82 – Registered 23 routes.  
2023-03-23 22:01:26,324 INFO event\_agent.py:56 – Report events to 10.244.0.6:34309  
2023-03-23 22:01:26,327 INFO event\_utils.py:136 – Monitor events logs modified after 1679632285.8408732 on /tmp/ray/session\_2023-03-23\_22-01-12\_707254\_14/logs/events, the source types are all.  
2023-03-23 22:01:26,649 WARNING agent.py:197 – Raylet is considered dead 3 X. If it reaches to 5, the agent will kill itself. Parent: None, parent\_gone: True, init\_assigned\_for\_parent: False, parent\_changed: False.  
2023-03-23 22:01:27,052 WARNING agent.py:197 – Raylet is considered dead 4 X. If it reaches to 5, the agent will kill itself. Parent: None, parent\_gone: True, init\_assigned\_for\_parent: False, parent\_changed: False.  
2023-03-23 22:01:27,458 WARNING agent.py:197 – Raylet is considered dead 5 X. If it reaches to 5, the agent will kill itself. Parent: None, parent\_gone: True, init\_assigned\_for\_parent: False, parent\_changed: False.  
2023-03-23 22:01:27,459 ERROR agent.py:249 – Raylet is terminated: ip=10.244.0.6, id=1c909ca6360bf4cabdbb6fc9f7e5192657f6263fef071fc66ccf892c. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:  
[state-dump] UNKNOWN - 2 total (2 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] InternalPubSubGcsService.grpc\_client.GcsSubscriberCommandBatch - 2 total (1 active), CPU time: mean = 1.593 ms, total = 3.185 ms  
[state-dump] NodeInfoGcsService.grpc\_client.RegisterNode - 1 total (0 active), CPU time: mean = 67.051 ms, total = 67.051 ms  
[state-dump] NodeInfoGcsService.grpc\_client.GetInternalConfig - 1 total (0 active), CPU time: mean = 1.311 s, total = 1.311 s  
[state-dump] ObjectManager.UpdateAvailableMemory - 1 total (0 active), CPU time: mean = 272.750 us, total = 272.750 us  
[state-dump] NodeManagerService.grpc\_server.GetResourceLoad - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] MemoryMonitor.CheckIsMemoryUsageAboveThreshold - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] InternalPubSubGcsService.grpc\_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] NodeManager.deadline\_timer.record\_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] NodeManagerService.grpc\_server.RequestResourceReport - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] NodeInfoGcsService.grpc\_client.GetAllNodeInfo - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] RayletWorkerPool.deadline\_timer.kill\_idle\_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] NodeManager.deadline\_timer.debug\_state\_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] NodeManager.deadline\_timer.flush\_free\_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s  
[state-dump] DebugString() time ms: 1  
[state-dump]  
[state-dump]  
[2023-03-23 22:01:21,187 I 229 229] (raylet) accessor.cc:590: Received notification for node id = 1c909ca6360bf4cabdbb6fc9f7e5192657f6263fef071fc66ccf892c, IsAlive = 1  
[2023-03-23 22:01:23,106 I 229 229] (raylet) accessor.cc:590: Received notification for node id = cc23449b1f256d08bd9714766584bb34c9cab61d1fc0352ee59cd884, IsAlive = 1  
[2023-03-23 22:01:26,308 I 229 229] (raylet) agent\_manager.cc:40: HandleRegisterAgent, ip: 10.244.0.6, port: 57134, id: 424238335

I didn’t find anything worthwhile in gcs\_server.out

Appreciate any help on this!

---

<div class="post-metadata">

**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\
**Post date:** [March 24, 2023, 7:04am UTC](https://discuss.ray.io/t/head-and-worked-node-dies-after-few-seconds/9878/2 "2023-03-24T07:04:33Z")

</div>

@VD23 Just curious. Why are you using docker on a local machine?

Can’t you follow the instructions on getting started [here](https://docs.ray.io/en/master/ray-overview/getting-started.html).

What type is your local machine?

---

<div class="post-metadata">

**Author:** ![VD23](https://avatars.discourse-cdn.com/v4/letter/v/e8c25b/32.png) [@VD23](https://discuss.ray.io/u/VD23)\
**Post date:** [March 24, 2023, 4:56pm UTC](https://discuss.ray.io/t/head-and-worked-node-dies-after-few-seconds/9878/3 "2023-03-24T16:56:49Z")

</div>

Thanks for your response @Jules_Damji. I use Mac with M1 chip. I did test the local ray cluster setup and was able to run a sample job. But, I’d eventually want to run Ray jobs on EKS, hence wanted to test on my local kubernetes cluster first.

Are there any known issues with using Ray cluster on docker ?

---

<div class="post-metadata">

**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\
**Post date:** [March 24, 2023, 5:09pm UTC](https://discuss.ray.io/t/head-and-worked-node-dies-after-few-seconds/9878/4 "2023-03-24T17:09:51Z")

</div>

@VD23 Not that I know of. Many Ray users use Ray on K8s.  
[Managed Kubernetes services — Ray 2.3.0](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/k8s-cluster-setup.html#setting-up-an-eks-cluster-aws).

Give it a try on AWS, if you have access to it, and let us know of any issues