# Gcs\_rpc\_client.h:179: Failed to connect to GCS at address 192.168.85.116:6379 within 5 seconds **URL:** \ **Category:** Configure Algorithm, Training, Evaluation, Scaling\ **Created:** [January 23, 2025, 8:14pm UTC](https://discuss.ray.io/t/gcs-rpc-client-h-failed-to-connect-to-gcs-at-address-192-168-85-116-6379-within-5-seconds/21593 "2025-01-23T20:14:19Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![tao\_sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tao_sun/32/7570_2.png) [@tao\_sun](https://discuss.ray.io/u/tao_sun)\ **Post date:** [January 23, 2025, 8:14pm UTC](https://discuss.ray.io/t/gcs-rpc-client-h-failed-to-connect-to-gcs-at-address-192-168-85-116-6379-within-5-seconds/21593/1 "2025-01-23T20:14:19Z")
The Ray cluster runs on Kubernetes (k8s). When I initialize Ray locally and try to submit code to the Ray cluster, it results in an error. However, submitting tasks directly from within the Ray pod works fine. The error message contains 192.168.85.116:6379, which is the IP address of the pod, but my local connection specifies the Ray cluster IP and NodePort. I don’t understand why the pod IP is thrown after submitting locally. After restarting the Ray pod, everything works normally for a while, but the same issue reoccurs after some time. Currently, I can only keep restarting Ray. Is there any way to resolve this? The error message is as follows: gcs\_rpc\_client.h:179: Failed to connect to GCS at address 192.168.85.116:6379 within 5 seconds. ---
**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\ **Post date:** [January 29, 2025, 12:01am UTC](https://discuss.ray.io/t/gcs-rpc-client-h-failed-to-connect-to-gcs-at-address-192-168-85-116-6379-within-5-seconds/21593/2 "2025-01-29T00:01:23Z")
Hi there tao\_sun, Someone else also recently ran into a similar issue, maybe you can see what they did and see if that’ll unblock you as well? They had to allow connections onto the port. > [@Certain Ray CLI commands complain about invalid GCS address](https://discuss.ray.io/t/certain-ray-cli-commands-complain-about-invalid-gcs-address/21565/6): > > Thanks for pointing me in the right direction Christina! I had to allow connections on port 6379 for ray status --address 10.212.154.239:6379 to work Curiously, I also had to allow connections on port 41823 for ray memory --address 10.212.154.239:6379 to work. However, it seems this port is chosen randomly at start, so after restarting my cluster the command failed again. Do you know perhaps which of the ray start options mentioned [here](https://docs.ray.io/en/latest/ray-core/configure.html#ports-configurations) would control this port so I can make it constant? I sho… Christy ---
**Author:** ![tao\_sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tao_sun/32/7570_2.png) [@tao\_sun](https://discuss.ray.io/u/tao_sun)\ **Post date:** [February 11, 2025, 2:44am UTC](https://discuss.ray.io/t/gcs-rpc-client-h-failed-to-connect-to-gcs-at-address-192-168-85-116-6379-within-5-seconds/21593/3 "2025-02-11T02:44:34Z")
This does not solve my issue. My Ray server and the client that initiates tasks are not in the same Kubernetes cluster. At the time of Ray initialization, my client was functioning normally, using the host machine’s IP and NodePort. However, after some time, for reasons unknown, my client changes the Ray server’s IP to the Pod IP and port, which results in connection failures. ---
**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\ **Post date:** [February 11, 2025, 8:22pm UTC](https://discuss.ray.io/t/gcs-rpc-client-h-failed-to-connect-to-gcs-at-address-192-168-85-116-6379-within-5-seconds/21593/5 "2025-02-11T20:22:27Z")
Hi tao, thanks for replying! After doing some thinking, can you let me know if you’ve tried these things out? ### **1. Pod IP vs. NodePort Confusion** - When you connect to Ray using the **NodePort** , it should keep that connection. - However, if Ray later tries to connect using the **Pod IP** , it might be because the GCS (Global Control Store) or another component restarted, causing Ray to resolve the IP differently. ### **2. GCS & Redis Issues** - If GCS is backed by **Redis** and it restarts, it may change the stored IP address. - Try using a fully qualified domain name (FQDN) instead of an IP address so that all nodes consistently resolve to the correct GCS instance ### **3. Consider Using KubeRay** - If you aren’t already using [**KubeRay**](https://github.com/ray-project/kuberay), consider switching to it. - KubeRay improves **fault tolerance** and **lifecycle management** for Ray components in Kubernetes! ### **4. Increase Reconnect Timeout** - May you can try setting the environment variable `RAY_gcs_rpc_server_reconnect_timeout_s` to a higher value, so this allows Ray nodes more time to **reconnect** after a GCS restart. ### **5. Check Your Network Configuration** - Verify that **NodePort** is correctly set and that your local machine can **persistently connect** to the Ray cluster. - Run `kubectl get services` to ensure the correct **external IP** is used. Here’s some docs that might be helpful: **Docs:** - [User Guides — Ray 2.52.0](https://docs.ray.io/en/latest/ray-core/user-guide.html#user-guides) - [GCS Fault Tolerance — Ray 2.52.0](https://docs.ray.io/en/latest/ray-core/fault_tolerance/gcs.html#setting-up-redis) - [Troubleshooting | Anyscale Docs](https://docs.anyscale.com/resources/troubleshooting) - [Common Issues — Ray 2.52.0](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/general-debugging.html#general-debugging) ---
**Author:** ![tao\_sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tao_sun/32/7570_2.png) [@tao\_sun](https://discuss.ray.io/u/tao_sun)\ **Post date:** [February 12, 2025, 1:20am UTC](https://discuss.ray.io/t/gcs-rpc-client-h-failed-to-connect-to-gcs-at-address-192-168-85-116-6379-within-5-seconds/21593/6 "2025-02-12T01:20:20Z")
Thank you for your professional reply and suggestions. We are currently using KubeRay. If the parsing error is really caused by GCS, I don’t think I can solve it. I can try to increase the RAY\_gcs\_rpc\_server\_reconnect\_timeout\_s to increase the connection duration.