With GCS FT enabled (external Redis/Valkey + a fixed externalStorageNamespace), why does restarting the head cause the cluster to “forget” the existing workers? Specifically: when the workers reconnect, are they still tied to an old session ID, and does the new head treat them as invalid/stale—so they get marked dead and the autoscaler deletes/recreates them?
Is it correct that GCS FT restores cluster metadata (e.g., Serve config, actor tables), but does not preserve/restore the live raylet session, so every head restart creates a new session ID?
When a worker reconnects, what identifier is checked to decide it belongs to the current session (session ID/epoch/token), and where is that stored and validated (on disk vs GCS/Redis)?
Is there any supported setting or mode that lets a restarted head adopt existing workers (avoid autoscaler pruning), or is worker recycling after head restart the expected behavior?
If this is expected, can you point us to the relevant documentation or code path that explains “session mismatch → node marked dead → autoscaler prunes”?
With GCS Fault Tolerance (FT) enabled (using external Redis/Valkey and a fixed externalStorageNamespace), restarting the head node does not preserve the live raylet session; each head restart generates a new session ID. When workers reconnect, they present their old session ID, which the new head treats as invalid—this results in the workers being marked as dead, and the autoscaler deletes and recreates them. This is expected behavior: GCS FT restores cluster metadata (like Serve config and actor tables), but not the live session or worker liveness; every head restart is effectively a new cluster session from the perspective of the raylets and workers. The session ID/token is stored in the GCS backend (e.g., Redis or RocksDB) and is validated during worker reconnection; mismatches cause the worker to be rejected as stale. There is currently no supported mode that allows a restarted head to adopt existing workers and avoid autoscaler pruning—worker recycling after head restart is the intended and documented behavior.
For documentation and code references: see the Node Fault Tolerance docs, which state that “when a raylet process fails, the corresponding node will be marked as dead and is treated the same as a node failure. Each raylet is associated with a unique id, so even if the raylet restarts on the same physical machine, it’ll be treated as a new raylet/node to the Ray cluster.” Also, REP-64 and this PR clarify that GCS FT provides persistence for cluster metadata, but not for live worker sessions. The session mismatch and node pruning logic is visible in the Raylet and GCS code, and is discussed in this issue and this discourse thread.
Would you like a step-by-step breakdown of the session validation and worker pruning code path?
Is any fix or change being worked on to allow head node restart to retain/adopt existing workers or is worker recycling after head node restart definitively the intended long-term behavior?
Can you confirm that with GCS FT enabled a head-node restart (due to repave activity) creates a new session ID, so existing workers reconnect with the old session, get treated as stale/invalid, are marked dead, and then the autoscaler prunes/recreates them? If that’s the intended behavior, could you share the relevant documentation and/or code path (files/functions) that drives the “session mismatch → node dead → autoscaler prune” flow so we can design around it?
Worker recycling after head node restart is currently the intended and expected long-term behavior in Ray, even with GCS FT enabled. There is no indication in the official documentation or recent PRs/issues that a change is planned to allow a restarted head node to adopt existing workers; the design treats each head restart as a new session, and existing workers with the old session ID are marked dead and pruned by the autoscaler. This is confirmed in REP-64 and Ray’s node fault tolerance docs.
The session mismatch and pruning flow is as follows: after a head node restart, a new session ID is generated (node.py), and when workers reconnect with the old session, they are treated as stale (node_manager.cc), marked dead, and the autoscaler then prunes and recreates them (autoscaler.py). This is also discussed in Ray Issue #64349.
Would you like a step-by-step code path breakdown for the session validation and pruning logic?