GCS Fault Tolerance: Worker pod restart during head pod replacement due to planned node repave

2

With GCS Fault Tolerance (FT) enabled (using external Redis/Valkey and a fixed externalStorageNamespace), restarting the head node does not preserve the live raylet session; each head restart generates a new session ID. When workers reconnect, they present their old session ID, which the new head treats as invalid—this results in the workers being marked as dead, and the autoscaler deletes and recreates them. This is expected behavior: GCS FT restores cluster metadata (like Serve config and actor tables), but not the live session or worker liveness; every head restart is effectively a new cluster session from the perspective of the raylets and workers. The session ID/token is stored in the GCS backend (e.g., Redis or RocksDB) and is validated during worker reconnection; mismatches cause the worker to be rejected as stale. There is currently no supported mode that allows a restarted head to adopt existing workers and avoid autoscaler pruning—worker recycling after head restart is the intended and documented behavior.

For documentation and code references: see the Node Fault Tolerance docs, which state that “when a raylet process fails, the corresponding node will be marked as dead and is treated the same as a node failure. Each raylet is associated with a unique id, so even if the raylet restarts on the same physical machine, it’ll be treated as a new raylet/node to the Ray cluster.” Also, REP-64 and this PR clarify that GCS FT provides persistence for cluster metadata, but not for live worker sessions. The session mismatch and node pruning logic is visible in the Raylet and GCS code, and is discussed in this issue and this discourse thread.

Would you like a step-by-step breakdown of the session validation and worker pruning code path?

Sources:

Hint: Mention @Herald in the post for followups.