# GCS FT + KubeRay: worker restart cascade during head pod replacement (planned node repave) - is this the intended contract?
**URL:** https://discuss.ray.io/t/gcs-ft-kuberay-worker-restart-cascade-during-head-pod-replacement-planned-node-repave-is-this-the-intended-contract/23595
**Category:** Uncategorized
**Created:** [August 2, 2026, 2:39am UTC](https://discuss.ray.io/t/gcs-ft-kuberay-worker-restart-cascade-during-head-pod-replacement-planned-node-repave-is-this-the-intended-contract/23595 "2026-08-02T02:39:53Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@getarbind](https://discuss.ray.io/u/getarbind)
#### Post date: [August 2, 2026, 2:39am UTC](https://discuss.ray.io/t/gcs-ft-kuberay-worker-restart-cascade-during-head-pod-replacement-planned-node-repave-is-this-the-intended-contract/23595/1 "2026-08-02T02:39:54Z")
During a planned EKS node repave (cordon/drain), our Ray head pods (KubeRay RayService/RayCluster with GCS Fault Tolerance enabled using external Valkey/Redis over TLS) were evicted and recreated. Even though the RayCluster spec did not change (no upgrade/new cluster), workers showed a repeatable two-step disruption: first, while the head was down, worker containers running **`ray start --block --address=:6379`** exited and restarted in a loop instead of waiting; second, after the head came back and workers successfully reconnected, the Ray autoscaler on the new head began deleting the “old” worker pods one-by-one and recreating them, causing a full worker recycle cascade.
We’re trying to understand if this is the intended behavior/contract of GCS FT with KubeRay (i.e., metadata recovery only) or if workers are expected to rejoin cleanly across head restart without being pruned. We can share exact Ray/KubeRay versions and logs/YAML if needed.
**Questions**
1. With **`RAY_gcs_rpc_server_reconnect_timeout_s=300`** , why does **`ray start --block`** on workers exit within ~60s when GCS/head is unreachable—does that timeout apply only to reconnect (not initial connect), and is there a knob for initial connect retry?
2. After head restart, why does the new head’s autoscaler classify reconnecting workers as “workers to be deleted” and delete/recreate them—does head restart always create a new session/epoch even with GCS FT?
3. Is worker rejoin across head restart _supported_ with GCS FT, or is the expected contract that GCS FT only restores metadata and worker recycling after head restart is normal/expected?
4. If this is expected, is there documentation (or a code-path reference) for the reconnect-timeout semantics and the autoscaler stale-worker pruning logic post head restart?
---
### Author:  [@RunLLM](https://discuss.ray.io/u/RunLLM)
#### Post date: [August 2, 2026, 2:40am UTC](https://discuss.ray.io/t/gcs-ft-kuberay-worker-restart-cascade-during-head-pod-replacement-planned-node-repave-is-this-the-intended-contract/23595/2 "2026-08-02T02:40:10Z")
1. **Worker exit on initial connect:** The `RAY_gcs_rpc_server_reconnect_timeout_s` environment variable only controls the timeout for reconnecting to GCS after a connection is lost, not the initial connection. On initial startup, workers use a separate (hardcoded) connect timeout, which defaults to 60 seconds. There is currently no documented environment variable or Ray config to extend the initial connect timeout; only reconnects are governed by `RAY_gcs_rpc_server_reconnect_timeout_s` ([source](https://github.com/ray-project/ray/blob/releases/2.47.1/doc/source/ray-core/fault_tolerance/gcs.rst), [source](https://github.com/ray-project/ray/blob/master/src/ray/common/ray_config_def.h), [source](https://github.com/ray-project/ray/blob/master/python/ray/_private/node.py)).
2. **Worker pruning after head restart:** With GCS fault tolerance enabled, Ray is designed to recover cluster metadata from Redis/Valkey, but the contract is not full cluster continuity—worker nodes are not guaranteed to persist across head restarts. After a head pod restart, the autoscaler may classify existing workers as stale and delete/recreate them, especially if their node IDs or registration epochs do not match the new head’s view of the cluster. This is expected behavior: GCS FT restores metadata, but worker recycling after head restart is normal and documented as such ([source](https://github.com/ray-project/ray/issues/20498), [source](https://github.com/ray-project/ray/issues/64349), [source](https://github.com/ray-project/ray/blob/releases/2.47.1/doc/source/ray-core/fault_tolerance/gcs.rst)). There is no guarantee that workers will rejoin cleanly across a head restart; the main goal of GCS FT is to preserve cluster metadata, not to provide seamless in-place worker continuity.
Would you like a step-by-step breakdown of the relevant code paths or more detail on the autoscaler’s stale-worker logic?
Sources:
- [Ray GCS Fault Tolerance documentation](https://github.com/ray-project/ray/blob/releases/2.47.1/doc/source/ray-core/fault_tolerance/gcs.rst)
- [Ray issue: Zombie workers and autoscaler pruning](https://github.com/ray-project/ray/issues/64349)
- [https://github.com/ray-project/ray/blob/master/src/ray/common/ray\_config\_def.h](https://github.com/ray-project/ray/blob/master/src/ray/common/ray_config_def.h)
- [https://github.com/ray-project/ray/blob/master/python/ray/\_private/node.py](https://github.com/ray-project/ray/blob/master/python/ray/_private/node.py)
- [https://github.com/ray-project/ray/issues/20498](https://github.com/ray-project/ray/issues/20498)
Hint: Mention @Herald in the post for followups.
---
### Author:  [@getarbind](https://discuss.ray.io/u/getarbind)
#### Post date: [August 2, 2026, 2:52am UTC](https://discuss.ray.io/t/gcs-ft-kuberay-worker-restart-cascade-during-head-pod-replacement-planned-node-repave-is-this-the-intended-contract/23595/3 "2026-08-02T02:52:19Z")
with **GCS FT enabled (external Redis/Valkey + fixed `externalStorageNamespace`)**, why does a head restart effectively “lose” the existing workers: do reconnecting workers present a **pre-restart session identity/token** that the new head **cannot adopt** , causing the head to mark them dead and the autoscaler to delete/recreate them?
- Is it correct that **GCS FT restores metadata** (Serve config/actor tables/etc.) but **does not restore/adopt the prior live raylet session** , so a new head always starts a new session ID?
- When workers reconnect, what exact identifier (session name/epoch/token) is used to decide they “belong” to the current cluster session, and where is it stored/validated?
- Is there any supported mode/flag/config where a restarted head can **accept/adopt existing worker raylets** (avoid autoscaler pruning), or is worker churn on head restart the expected contract?
- If this behavior is expected, can you point to the **doc or code path** that defines “session mismatch → node marked dead → autoscaler prunes”?