# Connecting to Ray cluster on Kubernetes from outside the cluster

**URL:** <https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364>\
**Category:** Kubernetes\
**Created:** [March 23, 2021, 3:36pm UTC](https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364 "2021-03-23T15:36:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 23, 2021, 3:36pm UTC](https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364/1 "2021-03-23T15:36:09Z")

</div>

I was trying to follow documentation [Deploying on Kubernetes — Ray v2.0.0.dev0](https://docs.ray.io/en/master/cluster/kubernetes.html#ray-k8s-run).  
Created Ray cluster on Kind cluster locally. Used both - ray up and operator approaches. Operator works great, for ray up it mostly works, but once in a while, it does not process ns creation and consequently fails. This behavior seems to be random.  
After cluster is created, Using Ray Client to connect from within the Kubernetes cluster works fine. I can look at the log and see the execution.  
But when I tried Using Ray Client to connect from outside the Kubernetes cluster connection always fails. I see that `Handling connection for 10001`, but the result is:  
Traceback (most recent call last):  
File “/Users/boris/Projects/RayOnKind/src/localPython.py”, line 55, in   
ray.util.connect(f"127.0.0.1:{LOCAL\_PORT}")  
File “/usr/local/lib/python3.7/site-packages/ray/util/client\_connect.py”, line 26, in connect  
conn\_str, secure=secure, metadata=metadata, connection\_retries=3)  
File “/usr/local/lib/python3.7/site-packages/ray/util/client/ **init**.py”, line 57, in connect  
connection\_retries=connection\_retries)  
File “/usr/local/lib/python3.7/site-packages/ray/util/client/worker.py”, line 120, in **init**  
raise ConnectionError(“ray client connection timeout”)  
ConnectionError: ray client connection timeout

To ensure Python version I am using image rayproject/ray:b3a717-py37-cpu.  
which is the same as my current:

python3 --version  
Python 3.7.10

---

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 23, 2021, 3:48pm UTC](https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364/2 "2021-03-23T15:48:30Z")

</div>

When I am running the same think in the notebook, I get a bit more info:

## 2021-03-23 10:45:49,624 INFO worker.py:113 – Waiting for Ray to become ready on the server, retry in 5s… 2021-03-23 10:45:59,633 INFO worker.py:113 – Waiting for Ray to become ready on the server, retry in 10s… 2021-03-23 10:46:14,642 INFO worker.py:113 – Waiting for Ray to become ready on the server, retry in 15s…

ConnectionError Traceback (most recent call last)  
 in   
1 #ray.init(ignore\_reinit\_error=True)  
2 #ray.init(address=‘127.0.0.1:10001’, \_redis\_password=‘5241590000000000’)  
----\> 3 ray.util.connect(‘127.0.0.1:10001’)

/usr/local/lib/python3.7/site-packages/ray/util/client\_connect.py in connect(conn\_str, secure, metadata, connection\_retries)  
24 # the correct metadata  
25 return ray.connect(  
—\> 26 conn\_str, secure=secure, metadata=metadata, connection\_retries=3)  
27  
28

/usr/local/lib/python3.7/site-packages/ray/util/client/ **init**.py in connect(self, conn\_str, secure, metadata, connection\_retries)  
55 secure=secure,  
56 metadata=metadata,  
—\> 57 connection\_retries=connection\_retries)  
58 self.api.worker = self.client\_worker  
59 return self.client\_worker.connection\_info()

/usr/local/lib/python3.7/site-packages/ray/util/client/worker.py in **init** (self, conn\_str, secure, metadata, connection\_retries)  
118 # up our retries and should error back to the user.  
119 if not ray\_ready:  
 → 120 raise ConnectionError(“ray client connection timeout”)  
121  
122 # Initialize the streams to finish protocol negotiation.

ConnectionError: ray client connection timeout

---

<div class="post-metadata">

**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\
**Post date:** [March 23, 2021, 7:05pm UTC](https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364/3 "2021-03-23T19:05:29Z")

</div>

Hey @Dmitri and @Ameer_Haj_Ali, could you take a look at this?

---

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 23, 2021, 8:47pm UTC](https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364/4 "2021-03-23T20:47:25Z")

</div>

Can it be because pip install ray install ray 1.2, while the cluster is using much newere version?

---

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 23, 2021, 10:51pm UTC](https://discuss.ray.io/t/connecting-to-ray-cluster-on-kubernetes-from-outside-the-cluster/1364/5 "2021-03-23T22:51:22Z")

</div>

And after reinstalling Ray to v2 the problem is fixed
