[Serve][Core] After Head node restart with GCS Redis, replicas on some nodes stay RECOVERING, get force-killed, and their GPUs are never freed (Ray 2.55)

Uncategorized
1

We use Ray 2.55.0 on KubeRay with GCS fault tolerance (external Redis) and RAY_gcs_rpc_server_reconnect_timeout_s=900. After the head pod was repaved/replaced (about a 10-minute gap), the new Serve controller couldn’t recover the replicas on 2 of 6 worker nodes. It force-killed them after 30 seconds, but their GPUs stayed marked as in use, so replacement replicas couldn’t be scheduled until we recreated the whole RayCluster.

What happened:

  1. A node repave evicted the head pod. Worker pods and replicas kept running.
  2. About 10 minutes later, a new head started. The controller found all replicas in GCS and marked them RECOVERING.
  3. On 2 worker nodes, every replica stayed RECOVERING for the full 30 seconds. On the other 4 nodes, almost all replicas recovered normally.
  4. At exactly 30 seconds, the controller logged Stopping … (currently ReplicaState.RECOVERING) and force-killed them with no grace period.
  5. The controller tried to add replacement replicas, but Ray still counted the GPUs on those 2 nodes as used. The replacements were never scheduled.

Questions

  1. Why would replicas that were running fine be restarted after the head node comes back?

  2. Why are only the replicas on some nodes affected, while other nodes recover normally?

  3. After those replicas are stopped, why aren’t their GPUs released? Is this a known issue in 2.55, and is it fixed in a newer version?

  4. Is there a setting to give replicas more time to recover after a head restart, instead of stopping them after about 30 seconds?

  5. Is there a way to free the stuck GPUs without recreating the whole RayCluster?

  6. What is the recommended setup to survive head node replacement on KubeRay? For example: a PodDisruptionBudget for the head, or specific Ray or Serve settings.

Environment
Ray 2.55.0, KubeRay, GCS FT with external Redis
RAY_gcs_rpc_server_reconnect_timeout_s=900

2

This is a known issue in Ray 2.55.0 and similar versions: after a head pod replacement with GCS fault tolerance, some replicas may fail to recover and are force-killed by the Serve controller after a fixed timeout (default 30s), but their resources (e.g., GPUs) remain marked as in use, blocking replacement replicas from being scheduled. This is due to stale leases or orphaned workers on the affected nodes, which the new GCS does not properly clean up after a failover. The issue is not deterministic—some nodes recover fine, others do not—likely due to timing or state inconsistencies during the reconnection window. This behavior and its impact on resource leakage are documented in recent Ray issues and are not fully resolved in 2.55.0; newer versions have partial fixes, but the problem may persist in some scenarios (see details, related discussion).

There is currently no Ray or Serve setting to extend the replica recovery timeout beyond the hardcoded 30 seconds, nor a built-in way to force-release stuck GPUs except by manually killing the idle worker processes or recreating the RayCluster. The recommended setup for head node replacement is to use GCS FT with a sufficiently high RAY_gcs_rpc_server_reconnect_timeout_s (as you have), and to protect the head pod with a PodDisruptionBudget to minimize unnecessary evictions. However, even with these precautions, resource leaks can still occur after head failover. Manual intervention (e.g., killing idle workers holding resources) is sometimes required until a more robust fix is available in Ray (see issue). Would you like a step-by-step breakdown of the recovery process and mitigation options?

Sources:

Hint: Mention @Herald in the post for followups.