# Remote worker nodes only alive for 30 seconds **URL:** \ **Category:** Ray Clusters\ **Created:** [November 22, 2022, 3:16pm UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394 "2022-11-22T15:16:31Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![bananajoe182](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bananajoe182/32/3452_2.png) [@bananajoe182](https://discuss.ray.io/u/bananajoe182)\ **Post date:** [November 22, 2022, 3:16pm UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/1 "2022-11-22T15:16:31Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. Hi, I’m trying to setup a small on premise cluster of high end GPU machines. I managed to connect the worker nodes to the head node but once they appear on the dashboard they are only alive for ~ 30 seconds. Also it is not possible to access their logs from the dashboard. I created firewall inbound rules for python, ray, raylet and gcs server executables to allow all port connections on both the head node and the worker machines. My setup: Windows 10 Pro Python 3.7.9 Ray 2.1.0 raylet.out ```auto [state-dump] Event stats: [state-dump] PeriodicalRunner.RunFnPeriodically - 8 total (1 active, 1 running), CPU time: mean = 252.338 us, total = 2.019 ms [state-dump] UNKNOWN - 3 total (3 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] NodeInfoGcsService.grpc_client.GetInternalConfig - 1 total (0 active), CPU time: mean = 190.864 ms, total = 190.864 ms [state-dump] NodeInfoGcsService.grpc_client.RegisterNode - 1 total (0 active), CPU time: mean = 280.800 us, total = 280.800 us [state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] ObjectManager.UpdateAvailableMemory - 1 total (0 active), CPU time: mean = 7.200 us, total = 7.200 us [state-dump] NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] NodeManager.deadline_timer.record_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] RayletWorkerPool.deadline_timer.kill_idle_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] NodeManager.deadline_timer.flush_free_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s [state-dump] DebugString() time ms: 1 [state-dump] [state-dump] [2022-11-18 15:10:14,582 I 14604 2452] (raylet.exe) accessor.cc:608: Received notification for node id = 8cc668ae845aca22294702a29e89487ae7d358b4bf7d5ba5d59ff9fa, IsAlive = 1 [2022-11-18 15:10:14,582 I 14604 2452] (raylet.exe) accessor.cc:608: Received notification for node id = d95f9e10cddcc512ed19448bcfa76c6660cdbaf48f60b7c5a481d044, IsAlive = 1 [2022-11-18 15:10:15,718 I 14604 2452] (raylet.exe) agent_manager.cc:40: HandleRegisterAgent, ip: 10.14.228.74, port: 59081, id: 15724 [2022-11-18 15:10:43,587 I 14604 2452] (raylet.exe) accessor.cc:608: Received notification for node id = 8cc668ae845aca22294702a29e89487ae7d358b4bf7d5ba5d59ff9fa, IsAlive = 0 [2022-11-18 15:10:43,591 C 14604 2452] (raylet.exe) node_manager.cc:1057: [Timeout] Exiting because this node manager has mistakenly been marked as dead by the GCS: GCS didn't receive heartbeats from this node for 30000 ms. This is likely because the machine or raylet has become overloaded. ***StackTrace Information*** unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown BaseThreadInitThunk RtlUserThreadStart ``` ---
**Author:** ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)\ **Post date:** [November 22, 2022, 6:35pm UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/2 "2022-11-22T18:35:07Z")
Do you mind posting your firewall configuration? In particular do you have bidirectional communication on these ports? [Configuring Ray — Ray 2.1.0](https://docs.ray.io/en/latest/ray-core/configure.html#ports-configurations) ---
**Author:** ![bananajoe182](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bananajoe182/32/3452_2.png) [@bananajoe182](https://discuss.ray.io/u/bananajoe182)\ **Post date:** [November 24, 2022, 9:54am UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/3 "2022-11-24T09:54:57Z")
Hi Alex, thanks for your time. The interesting thing is that I tried the same at my personal network at home (Same OS and Python env) and there it seems to work. The following settings where created automatically when the windows defender popup was accepted. So I copied these settings to our company machines but somehow I have the problem with the nodes being alive only a few seconds there. Here are snippets from my firewall Inbound rules : ![2022-11-24 10_56_26-Windows Defender Firewall with Advanced Security](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/4/47df9728bf7c7e68ef175ddf52a420b609d0f042.png) ![2022-11-24 10_56_58-Windows Defender Firewall with Advanced Security](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/3/33b502831a500193b9ead4d4ed8fbc1ce6618c7a.png) ![2022-11-24 10_57_21-Windows Defender Firewall with Advanced Security](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/c/cf40f3ff68e2daa510244602b317a5a8fb67a742.png) ---
**Author:** ![bananajoe182](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bananajoe182/32/3452_2.png) [@bananajoe182](https://discuss.ray.io/u/bananajoe182)\ **Post date:** [December 20, 2022, 1:43pm UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/4 "2022-12-20T13:43:28Z")
Hey, the problem is solved now. I identified 2 problems: 1. Firewall Inbound rules were overruled by Group Policy settings so the local settings were ignored 2. As I used a Python virtual env, I also had to create a rule for the base python 3.7 .exe as well, in addition to the venv python.exe. That was the reason that the logs were not accessible. ---
**Author:** ![varun\_raju](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/varun_raju/32/3948_2.png) [@varun\_raju](https://discuss.ray.io/u/varun_raju)\ **Post date:** [February 23, 2023, 12:35pm UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/6 "2023-02-23T12:35:26Z")
How did you solved the above mentioned problems? ---
**Author:** ![doryan607](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@doryan607](https://discuss.ray.io/u/doryan607)\ **Post date:** [April 10, 2023, 1:20am UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/7 "2023-04-10T01:20:27Z")
@bananajoe182 Hello, I have the same problem as you mentioned. I have a quick question about the solutions you presented. 1. What group policy settings did you edit? 2. Which rule should I create for the python 3.7 .exe and venv python.exe? (Group policy, inbound, etc.) Thank you for your help in advance! ---
**Author:** ![bananajoe182](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bananajoe182/32/3452_2.png) [@bananajoe182](https://discuss.ray.io/u/bananajoe182)\ **Post date:** [April 10, 2023, 7:53am UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/8 "2023-04-10T07:53:30Z")
Hi, You have to allow all Inbound Connections for the system python.exe as well es venv python.exe if you use venv. Additionally allow all for ray.exe and gsc\_server.exe. Do this for all worker nodes and the head node. Be sure if your machine is part of a company network that those rules are set as administrator so it overrules local user settings. If it still doesn’t work try adding the same for outbound connections. Hope that helps! ---
**Author:** ![MiJi](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@MiJi](https://discuss.ray.io/u/MiJi)\ **Post date:** [April 24, 2025, 8:34am UTC](https://discuss.ray.io/t/remote-worker-nodes-only-alive-for-30-seconds/8394/9 "2025-04-24T08:34:19Z")
Hi, I had a similar issue with the Azure VM as well and inside a docker container. The instantiation of the docker container needed to be allowed with the “host” network, otherwise the docker container uses it’s own isolated bridge. you can add the docker container to the ray head as a node, but will be vanished after 30 seconds. in a docker-compose, it is needed only to add the following line to the docker-compose.yml file: network\_mode: “host” Hope that helps!