Ray head gcs_server crashes with SIGSEGV (exit code -11) during ElastiCache/Valkey primary failover — GCS-FT does not survive Redis primary switchover

1

During a controlled chaos test that forces an ElastiCache (Valkey 8.1) primary→replica failover, the Ray head container’s gcs_server process crashes with signal 11 (SIGSEGV) approximately 10 seconds after the failover completes. This kills the entire Ray head pod (Kubernetes restarts it), causing ~30–60 s of Ray control-plane unavailability and load-test errors. We expected GCS-FT to transparently reconnect to the new primary via the ElastiCache master. DNS endpoint and continue.

Is Valkey 8.1 (ElastiCache) a supported GCS-FT backend at our Ray/KubeRay version? Are there known incompatibilities with the Valkey engine vs. Redis OSS ?

2

Valkey 8.1 is supported as a drop-in backend for Ray GCS fault tolerance, and recent community testing confirms that Ray’s GCS-FT works with Valkey using standard Redis commands (Discussion: support Valkey). However, Ray is not compatible with AWS ElastiCache Serverless Valkey due to missing command support, but ElastiCache for Valkey in cluster mode (with full command set) is reported to work. There are no known incompatibilities between Valkey OSS and Redis OSS for the commands Ray uses (HSET, HGET, HMGET, HDEL, HSCAN, INCRBY, PING, SCAN, DEL), but some managed/cloud flavors may restrict commands or have failover behaviors that can trigger Ray GCS crashes (see here).

The SIGSEGV after failover is a known Ray bug affecting both Redis and Valkey backends: when the primary fails over, Ray’s GCS process can dereference a null pointer before reconnecting, causing a crash and pod restart. This is not specific to Valkey and is being addressed in Ray PR #64204 and issue #53475. Upgrading to a Ray version with this fix will prevent the crash, but until then, GCS-FT recovery will involve a brief outage and restart. Would you like more detail on mitigation or upgrade paths?

Sources:

Hint: Mention @Herald in the post for followups.

3

I’m currently running Ray 2.55.0 (Python 3.11, CUDA 12.8) and I’m hitting the same SIGSEGV/GCS crash on Redis/Valkey failover described in [Ray Serve] GCS Segmentation Fault on failed Redis requests #53475.

It sounds like the fix is PR #64204 (“Avoid GCS crash on Redis connection loss in RedisResponseFn”), referenced as the fix candidate for this crash class (see the Ray repo pulse reference) (2 | github.com).

Question: Which released Ray version first includes PR #64204 (i.e., definitively resolves #53475)? If it’s not in a stable tag yet, what’s the recommended nightly version / commit SHA to use?

@Herald could you confirm the target release version for #64204?

4

As of now, PR #64204 (“Avoid GCS crash on Redis connection loss in RedisResponseFn”) is merged but has not yet been included in any stable Ray release tag. The fix is not present in Ray 2.55.0 or earlier stable versions. To use this fix, you need to install a Ray nightly wheel built after the merge date of PR #64204 (June 18, 2026). The recommended approach is to use the latest nightly wheel from the Ray S3 nightly wheels or a wheel built from a commit after June 18, 2026 (see PR #64204).

There is no official release tag containing this fix yet. To ensure you have the patch, use a nightly wheel with a commit SHA after the merge of #64204. Would you like instructions on how to install the latest nightly Ray wheel?

Sources:

Hint: Mention @Herald in the post for followups.

5

I need this info — yes, please share the exact install instructions for a Ray nightly wheel built after the PR #64204 merge (June 18, 2026), and how to verify I’m actually running a build that includes that patch (e.g., commit SHA/date check).

Context on my side:

  • Current: Ray 2.55.0
  • Env: Python 3.11, CUDA 12.8

Specifically, can you provide:

  1. The recommended nightly wheel identifier (date + commit SHA) that’s known-good for #64204

@Herald — can you confirm the best nightly build to use until this lands in a stable release tag?

6

It seems likely that to get the fix from PR #64204, you should install a Ray nightly wheel built after June 18, 2026. The nightly wheels are published daily and include the latest merged PRs. The recommended way is to use the latest nightly wheel, or a wheel with a commit SHA after the merge of #64204. The commit SHA for #64204 is b7e2e2e (see PR #64204).

Install instructions:

  1. Uninstall existing Ray:

    pip uninstall -y ray

  2. Install the latest Ray nightly wheel (Python 3.11, CUDA 12.8, Linux example):

    pip install -U "ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp311-cp311-manylinux2014_x86_64.whl"

  3. Verify the installed Ray version and commit SHA:

    import ray
print(ray.__version__)
print(ray.__commit__)
    • Ensure that ray.__commit__ is at or after b7e2e2e and the build date is after June 18, 2026.

If you want a specific nightly build, you can use a wheel from a specific commit:

pip install "https://s3-us-west-2.amazonaws.com/ray-wheels/master/b7e2e2e/ray-3.0.0.dev0-cp311-cp311-manylinux2014_x86_64.whl"

Replace the commit and wheel filename as needed for your environment.

Would you like more detail on how to find the exact wheel for your platform or how to check the commit in the wheel?

Sources:

Hint: Mention @Herald in the post for followups.