With GCS FT enabled (external Redis/Valkey + a fixed externalStorageNamespace), why does restarting the head cause the cluster to “forget” the existing workers? Specifically: when the workers reconnect, are they still tied to an old session ID, and does the new head treat them as invalid/stale—so they get marked dead and the autoscaler deletes/recreates them?
Is it correct that GCS FT restores cluster metadata (e.g., Serve config, actor tables), but does not preserve/restore the live raylet session, so every head restart creates a new session ID?
When a worker reconnects, what identifier is checked to decide it belongs to the current session (session ID/epoch/token), and where is that stored and validated (on disk vs GCS/Redis)?
Is there any supported setting or mode that lets a restarted head adopt existing workers (avoid autoscaler pruning), or is worker recycling after head restart the expected behavior?
If this is expected, can you point us to the relevant documentation or code path that explains “session mismatch → node marked dead → autoscaler prunes”?
With GCS Fault Tolerance (FT) enabled (using external Redis/Valkey and a fixed externalStorageNamespace), restarting the head node does not preserve the live raylet session; each head restart generates a new session ID. When workers reconnect, they present their old session ID, which the new head treats as invalid—this results in the workers being marked as dead, and the autoscaler deletes and recreates them. This is expected behavior: GCS FT restores cluster metadata (like Serve config and actor tables), but not the live session or worker liveness; every head restart is effectively a new cluster session from the perspective of the raylets and workers. The session ID/token is stored in the GCS backend (e.g., Redis or RocksDB) and is validated during worker reconnection; mismatches cause the worker to be rejected as stale. There is currently no supported mode that allows a restarted head to adopt existing workers and avoid autoscaler pruning—worker recycling after head restart is the intended and documented behavior.
For documentation and code references: see the Node Fault Tolerance docs, which state that “when a raylet process fails, the corresponding node will be marked as dead and is treated the same as a node failure. Each raylet is associated with a unique id, so even if the raylet restarts on the same physical machine, it’ll be treated as a new raylet/node to the Ray cluster.” Also, REP-64 and this PR clarify that GCS FT provides persistence for cluster metadata, but not for live worker sessions. The session mismatch and node pruning logic is visible in the Raylet and GCS code, and is discussed in this issue and this discourse thread.
Would you like a step-by-step breakdown of the session validation and worker pruning code path?