Exception: ray.init() called, but ray client is already connected

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

This error occurs when trying to deploy a ray-serve deployment to a ray cluster using (
ray.init(
address=‘ray://127.0.0.1:10001’,
namespace=“serve”,
runtime_env=runtime_env,
ignore_reinit_error=True
)

If I run one node locally and set
ray.init(
address=‘auto’,
namespace=“serve”,
runtime_env=runtime_env,
ignore_reinit_error=True
)
ray serve deployment works fine.

trace:
Traceback (most recent call last):
File “deploy.py”, line 50, in
ignore_reinit_error=True
File “/home/pnikulshin/projects/serve_robots/face_control/venv/lib/python3.7/site-packages/ray/util/client/init.py”, line 282, in init
ret = self.get_context().init(*args, **kwargs)
File “/home/pnikulshin/projects/serve_robots/face_control/venv/lib/python3.7/site-packages/ray/util/client/init.py”, line 195, in init
self.connect(“127.0.0.1:50051”)
File “/home/pnikulshin/projects/serve_robots/face_control/venv/lib/python3.7/site-packages/ray/util/client/init.py”, line 68, in connect
raise Exception(“ray.init() called, but ray client is already connected”)
Exception: ray.init() called, but ray client is already connected

Try to ray-1.13.0 and ray-2.0.0rc1, python3.7.7

Hi @hyrmec , sorry you’re running into this. Could you help us out with some short self-contained reproduction steps? Is it that you’re runningray start --head on your local machine, and then the line

ray.init(
address=‘ray://127.0.0.1:10001’,
namespace=“serve”,
runtime_env=runtime_env,
ignore_reinit_error=True
)

fails with that error? Or does it fail when you deploy a Serve deployment?

I have a cluster running in k8s, I have forwarded the ray ports to myself locally. I’m trying to deploy my ray-serve deployment to the cluster, which is well deployed locally.
I am using RuntimeEnv to deliver my code and environment to the cluster.

My code:
from fastapi import Request, Response, FastAPI, HTTPException
from ray import serve
from ray.util.client import ray
from ray.runtime_env import RuntimeEnv

app = FastAPI()
runtime_env = RuntimeEnv(
working_dir=‘https://path_to_my_repo.zip’,
pip=‘./requirements.txt’
)
ray.init(
#address=‘auto’,
address=‘ray://127.0.0.1:10001’,
namespace=“serve”,
runtime_env=runtime_env,
ignore_reinit_error=True
)
serve.start(detached=True)

@serve.deployment()
@serve.ingress(app)
class FaceControl:
def init(self):
self.face_recognizer = ‘my-robot’

@app.get("/compare_faces")
def compare_faces(self, main_url: str, target_url: str):
    return True

FaceControl.deploy()

I execute python deploy.py
and get Exception: ray.init() called, but ray client is already connected

Tell me how you deploy the ray serve code to the ray cluster

Thanks for the further details! Let me try to reproduce this; I don’t see anything wrong with the example you posted. In the meantime, you can try calling ray.shutdown() on the line before ray.init(), which may get rid of the “already connected” error, though I don’t yet understand how it could already be connected.

@architkulkarni The only successful ray-serve deploy option:

kubectl exec -it head_pod
git clone my repo
run deploy script

If RAY_ADDRESS != auto then there are problems either with access to the module or with something.

Glad you were able to find a workaround. Did ray.shutdown() not prevent the error? In general, ray.init("ray:// ...") should work when run locally to connect to a remote cluster.

In Ray 2.0, the recommended way to run Ray Serve in production on Kubernetes is to deploy a RayService on KubeRay: Deploying on Kubernetes — Ray 2.0.0. This might be helpful if you’re able to upgrade to Ray 2.0.