With GCS FT enabled (external Redis/Valkey + a fixed externalStorageNamespace), why does restarting the head cause the cluster to “forget” the existing workers? Specifically: when the workers reconnect, are they still tied to an old session ID, and does the new head treat them as invalid/stale—so they get marked dead and the autoscaler deletes/recreates them?
Is it correct that GCS FT restores cluster metadata (e.g., Serve config, actor tables), but does not preserve/restore the live raylet session, so every head restart creates a new session ID?
When a worker reconnects, what identifier is checked to decide it belongs to the current session (session ID/epoch/token), and where is that stored and validated (on disk vs GCS/Redis)?
Is there any supported setting or mode that lets a restarted head adopt existing workers (avoid autoscaler pruning), or is worker recycling after head restart the expected behavior?
If this is expected, can you point us to the relevant documentation or code path that explains “session mismatch → node marked dead → autoscaler prunes”?
With GCS Fault Tolerance (FT) enabled (using external Redis/Valkey and a fixed externalStorageNamespace), restarting the head node does not preserve the live raylet session; each head restart generates a new session ID. When workers reconnect, they present their old session ID, which the new head treats as invalid—this results in the workers being marked as dead, and the autoscaler deletes and recreates them. This is expected behavior: GCS FT restores cluster metadata (like Serve config and actor tables), but not the live session or worker liveness; every head restart is effectively a new cluster session from the perspective of the raylets and workers. The session ID/token is stored in the GCS backend (e.g., Redis or RocksDB) and is validated during worker reconnection; mismatches cause the worker to be rejected as stale. There is currently no supported mode that allows a restarted head to adopt existing workers and avoid autoscaler pruning—worker recycling after head restart is the intended and documented behavior.
For documentation and code references: see the Node Fault Tolerance docs, which state that “when a raylet process fails, the corresponding node will be marked as dead and is treated the same as a node failure. Each raylet is associated with a unique id, so even if the raylet restarts on the same physical machine, it’ll be treated as a new raylet/node to the Ray cluster.” Also, REP-64 and this PR clarify that GCS FT provides persistence for cluster metadata, but not for live worker sessions. The session mismatch and node pruning logic is visible in the Raylet and GCS code, and is discussed in this issue and this discourse thread.
Would you like a step-by-step breakdown of the session validation and worker pruning code path?
Is any fix or change being worked on to allow head node restart to retain/adopt existing workers or is worker recycling after head node restart definitively the intended long-term behavior?
Can you confirm that with GCS FT enabled a head-node restart (due to repave activity) creates a new session ID, so existing workers reconnect with the old session, get treated as stale/invalid, are marked dead, and then the autoscaler prunes/recreates them? If that’s the intended behavior, could you share the relevant documentation and/or code path (files/functions) that drives the “session mismatch → node dead → autoscaler prune” flow so we can design around it?
Worker recycling after head node restart is currently the intended and expected long-term behavior in Ray, even with GCS FT enabled. There is no indication in the official documentation or recent PRs/issues that a change is planned to allow a restarted head node to adopt existing workers; the design treats each head restart as a new session, and existing workers with the old session ID are marked dead and pruned by the autoscaler. This is confirmed in REP-64 and Ray’s node fault tolerance docs.
The session mismatch and pruning flow is as follows: after a head node restart, a new session ID is generated (node.py), and when workers reconnect with the old session, they are treated as stale (node_manager.cc), marked dead, and the autoscaler then prunes and recreates them (autoscaler.py). This is also discussed in Ray Issue #64349.
Would you like a step-by-step code path breakdown for the session validation and pruning logic?
We’re seeing behavior with Ray GCS Fault Tolerance enabled (external Redis/Valkey) where restarting the head makes the cluster “forget” existing workers. When workers reconnect, it looks like they may still present an old session ID, and the new head treats them as stale/invalid, so they get marked dead and then the autoscaler prunes/deletes and recreates them.
Can you please confirm if we have any existing issue / ticket / PR open for this exact behavior? If yes, could you share the link and current status?
Thanks in advance for confirming and sharing whether there’s an active fix in flight or if we should treat this as expected and design around it.
Hi Arbind, rather than add another opinion we built your scenario and measured it: KubeRay, GCS FT with external Redis, pinned externalStorageNamespace,
Ray 2.55.0, head alone on its own node, worker on a separate node.
The bot answers in this thread are wrong. We did a planned repave of the head’s node: cordon, drain, delete the node. The head pod moved to a different node, pulled its image cold, and was Ready 64 seconds later on a new pod IP. The worker pod was untouched throughout, RESTARTS 0, no GCS errors in its log. There is no session ID check and no rejection of reconnecting workers. A restarted head adopting its existing workers is the design, and we watched it work across a node replacement.
The only thing that recycled a worker was the reconnect budget. We repeated the test with the replacement head held out of the cluster. The worker survived 596 seconds untouched, then exited with code 1 at 607 seconds against a 600 second budget. So a worker tolerates the head being absent for its RAY_gcs_rpc_server_reconnect_timeout_s and no longer, and the exit is clean, not a crash.
Taken together, your workers dying means your head-absent window exceeded your worker budget. Ours was 64 seconds against 600 and nothing happened. Two numbers would confirm it: the wall-clock gap between the old head pod disappearing and the new one reaching Ready, and what your budget actually resolves to:
kubectl get pod -o jsonpath=‘{.spec.containers[0].env}’
That second one is easy to get backwards: KubeRay injects RAY_gcs_rpc_server_reconnect_timeout_s=600 onto worker pods only if you have not set it yourself, so any value in your spec replaces the 600 rather than adding to it.
If the two numbers above do not explain it, this settles which mechanism did:
kubectl get pod -o jsonpath=‘{.status.containerStatuses[0].lastState.terminated}’
kubectl logs -c ray-worker --previous | tail -50
Code 1 with a “Failed to connect to GCS within N seconds” line is point 2. Anything else is a mechanism we have not reproduced, and we would want to see it.
Hi @getarbind, apologies for the delay. I first saw your response yesterday afternoon.
Yes, you can make GCS FT and zero-downtime work together, and you are already running it. Unpinned is the supported configuration for both, and it costs nothing in fault tolerance. GCS FT is on because the cluster has a redis address. ExternalStorageNamespace only decides which key prefix the metadata sits under, and left unset KubeRay uses the RayCluster UID, a property of the RayCluster object rather than the head pod, so it survives any head restart. The official GCS FT walkthrough is itself unpinned. ray-cluster.external-redis.yaml sets no externalStorageNamespace, and that is the configuration the docs use to kill the GCS process and show the worker surviving.
Pinning only allows a brand new RayCluster to adopt a previous one’s metadata across a UID change. That is precisely what breaks a zero-downtime upgrade. Two RayClusters are alive at once, the new head reads the old cluster’s Serve metadata, and the operator concludes the applications are already running and shifts traffic before the new cluster is up. The two behaviors are mutually exclusive by construction, and unpinned is the one a RayService wants.
Hi @getarbind Following up on this, are you still seeing any issues wrt FT? If so, can you please provide the logs or the steps needed to reproduce? Thanks