During a planned EKS node repave (cordon/drain), our Ray head pods (KubeRay RayService/RayCluster with GCS Fault Tolerance enabled using external Valkey/Redis over TLS) were evicted and recreated. Even though the RayCluster spec did not change (no upgrade/new cluster), workers showed a repeatable two-step disruption: first, while the head was down, worker containers running ray start --block --address=<head-svc>:6379 exited and restarted in a loop instead of waiting; second, after the head came back and workers successfully reconnected, the Ray autoscaler on the new head began deleting the “old” worker pods one-by-one and recreating them, causing a full worker recycle cascade.
We’re trying to understand if this is the intended behavior/contract of GCS FT with KubeRay (i.e., metadata recovery only) or if workers are expected to rejoin cleanly across head restart without being pruned. We can share exact Ray/KubeRay versions and logs/YAML if needed.
Questions
With RAY_gcs_rpc_server_reconnect_timeout_s=300, why does ray start --block on workers exit within ~60s when GCS/head is unreachable—does that timeout apply only to reconnect (not initial connect), and is there a knob for initial connect retry?
After head restart, why does the new head’s autoscaler classify reconnecting workers as “workers to be deleted” and delete/recreate them—does head restart always create a new session/epoch even with GCS FT?
Is worker rejoin across head restart supported with GCS FT, or is the expected contract that GCS FT only restores metadata and worker recycling after head restart is normal/expected?
If this is expected, is there documentation (or a code-path reference) for the reconnect-timeout semantics and the autoscaler stale-worker pruning logic post head restart?
Worker exit on initial connect: The RAY_gcs_rpc_server_reconnect_timeout_s environment variable only controls the timeout for reconnecting to GCS after a connection is lost, not the initial connection. On initial startup, workers use a separate (hardcoded) connect timeout, which defaults to 60 seconds. There is currently no documented environment variable or Ray config to extend the initial connect timeout; only reconnects are governed by RAY_gcs_rpc_server_reconnect_timeout_s (source, source, source).
Worker pruning after head restart: With GCS fault tolerance enabled, Ray is designed to recover cluster metadata from Redis/Valkey, but the contract is not full cluster continuity—worker nodes are not guaranteed to persist across head restarts. After a head pod restart, the autoscaler may classify existing workers as stale and delete/recreate them, especially if their node IDs or registration epochs do not match the new head’s view of the cluster. This is expected behavior: GCS FT restores metadata, but worker recycling after head restart is normal and documented as such (source, source, source). There is no guarantee that workers will rejoin cleanly across a head restart; the main goal of GCS FT is to preserve cluster metadata, not to provide seamless in-place worker continuity.
Would you like a step-by-step breakdown of the relevant code paths or more detail on the autoscaler’s stale-worker logic?
with GCS FT enabled (external Redis/Valkey + fixed externalStorageNamespace), why does a head restart effectively “lose” the existing workers: do reconnecting workers present a pre-restart session identity/token that the new head cannot adopt, causing the head to mark them dead and the autoscaler to delete/recreate them?
Is it correct that GCS FT restores metadata (Serve config/actor tables/etc.) but does not restore/adopt the prior live raylet session, so a new head always starts a new session ID?
When workers reconnect, what exact identifier (session name/epoch/token) is used to decide they “belong” to the current cluster session, and where is it stored/validated?
Is there any supported mode/flag/config where a restarted head can accept/adopt existing worker raylets (avoid autoscaler pruning), or is worker churn on head restart the expected contract?
If this behavior is expected, can you point to the doc or code path that defines “session mismatch → node marked dead → autoscaler prunes”?