How can GCS Fault Tolerance (GCS FT) (backed by Redis/ElastiCache) coexist with RayService NewClusterWithIncrementalUpgrade without corrupting/overwriting cluster state when two head pods temporarily overlap?
We’re using RayService’s incremental upgrade strategy ( RayService Zero-Downtime Incremental Upgrades — Ray 2.56.0 ) to achieve zero/low-downtime upgrades without full blue/green resource duplication. We also want to enable GCS FT for head resilience. However, during upgrade we effectively have two head pods alive at the same time, and it appears both can write to the same Redis-backed GCS FT store. We’re seeing behavior that looks like the new head reads/loads the old head’s state during initialization, causing failures (potentially due to stale/incorrect cluster metadata being picked up).
Questions:
Is GCS FT supported/recommended with NewClusterWithIncrementalUpgrade, given the temporary “two heads” window?
If they are meant to work together, what mechanism prevents two heads from writing to the same FT backend (e.g., leader election, fencing, epochs/cluster-id isolation)?
What is the required configuration to isolate state between the old and new clusters during an incremental upgrade (e.g., per-cluster Redis namespace/prefix, unique cluster IDs, separate Redis DBs, separate ElastiCache endpoints)?
If isolation isn’t possible today, is the guidance to disable GCS FT during incremental upgrades, or to use a different RayService upgrade strategy when GCS FT is enabled?
GCS Fault Tolerance (GCS FT) with Redis/ElastiCache is not compatible with RayService’s NewClusterWithIncrementalUpgrade strategy unless you ensure strict state isolation between the old and new clusters. If both clusters (old and new) share the same Redis namespace or backend during the upgrade, both head pods can read/write the same GCS state, leading to corruption or failures, as you’ve observed. The Ray documentation explicitly warns that specifying the same ray.io/external-storage-namespace for both clusters can cause the new cluster to load stale or incorrect metadata, resulting in downtime or misbehavior during upgrades. There is no built-in fencing, leader election, or epoch mechanism in Ray GCS FT to prevent this overlap—isolation is your responsibility (RayService troubleshooting, KubeRay GCS FT guide).
Required configuration: To safely use GCS FT with incremental upgrades, you must ensure each RayCluster (old and new) uses a unique Redis namespace. The recommended approach is to remove the ray.io/external-storage-namespace annotation from the RayService CRD. When omitted, KubeRay automatically assigns a unique namespace (using the RayCluster’s UID), so each cluster’s GCS state is isolated in Redis. Alternatively, you can manually set a unique value for ray.io/external-storage-namespace for each cluster. Do not allow both clusters to share the same Redis DB/namespace or endpoint during the upgrade window. If you cannot guarantee this isolation, the guidance is to avoid enabling GCS FT during incremental upgrades or use a different upgrade strategy (RayService troubleshooting, KubeRay GCS FT guide).
Would you like a step-by-step example of the recommended configuration?
As RunLLM said, you can’t use externalStorageNamespace or ray.io/external-storage-namespace annotation; otherwise, you will have the issue you mentioned: the new head reads/loads the old head’s state during initialization.