# Ray client fails to reconnect

**URL:** <https://discuss.ray.io/t/ray-client-fails-to-reconnect/4167>\
**Category:** Uncategorized\
**Created:** [November 18, 2021, 1:31am UTC](https://discuss.ray.io/t/ray-client-fails-to-reconnect/4167 "2021-11-18T01:31:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tanvi\_Thakur](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tanvi_thakur/32/1776_2.png) [@Tanvi\_Thakur](https://discuss.ray.io/u/Tanvi_Thakur)\
**Post date:** [November 18, 2021, 1:31am UTC](https://discuss.ray.io/t/ray-client-fails-to-reconnect/4167/1 "2021-11-18T01:31:44Z")

</div>

Hey, trying to run a simple ML Model using ray which uses Ray Tune as well. We are using EKS cluster and ray-operator in order to use Ray. Apparently, when we try to run our ML model in jupyter notebook, it fails giving error:

Request can’t be sent because the Ray client has already been disconnected due to an error. Last exception: Failed to reconnect within the reconnection grace period (30s)

Any insights on this? We tried changing our cluster config as well in order to use high cpu and better ec2 instances, but it is still throwing an error. Is there a way we can increase the number of threads? (

---

<div class="post-metadata">

**Author:** ![samrogers226](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/samrogers226/32/580_2.png) [@samrogers226](https://discuss.ray.io/u/samrogers226)\
**Post date:** [November 18, 2021, 7:39pm UTC](https://discuss.ray.io/t/ray-client-fails-to-reconnect/4167/2 "2021-11-18T19:39:57Z")

</div>

Hi @Tanvi_Thakur,

Is it possible that you are opening too many clients to the cluster? This can be increased by setting the `RAY_CLIENT_SERVER_MAX_THREADS` environment variable on the server side. I believe by default it is 50.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![Tanvi\_Thakur](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tanvi_thakur/32/1776_2.png) [@Tanvi\_Thakur](https://discuss.ray.io/u/Tanvi_Thakur)\
**Post date:** [November 18, 2021, 8:09pm UTC](https://discuss.ray.io/t/ray-client-fails-to-reconnect/4167/3 "2021-11-18T20:09:58Z")

</div>

In rayclusters K8s crd, I tried changing the RAY\_gcs\_server\_rpc\_server\_thread\_num but the max it takes is 10 after that it gives error as head stops accepting connections. Where is this RAY\_CLIENT\_SERVER\_MAX\_THREADS variable set?