Ray Serve HTTP requests handling

Ray Libraries (Data, Train, Tune, Serve)
1

Ray clusters are deployed on several virtual machines (Linux Debian 11, Python 3.9.2, Ray versions are the same). The trained model is deployed on these clusters. I can’t send a request from my machine to the head node, but I can send a request inside the virtual machine to localhost:8000. The question is: how to open the Ray Serve server for external requests?

Also I tryed to send a remote request like this:

@ray.remote
def remote_request():
    print(requests.get("http://localhost:8000/some_prefix").json())

remote_request.remote()

And also didn’t got anything, but this warning:

WARNING long_poll.py:149 -- LongPollClient connection failed, shutting down.
2

@GolikovAndrey You will probably need to put a proxy NGIX or FastAPI server and have it route to the Serve deployment.

cc: @Sihan_Wang do we have a best practice example in the doc how to expose Ray Serve deployment as external services on OSS Ray?

3

Hi @GolikovAndrey, to rule out network problem, have you ever successfully sent requests from your machine to the other virtual machine? (E.g. AWS has some network default port rules to block outbound traffic, user has to enable specific port to receive traffic.)

Thanks
Sihan

4

Thanks for the answer guys @Jules_Damji, @Sihan_Wang!

We don’t work with cloud services like AWS or GCP, our system admins deploy virtual machines on our servers.
I found solution of my problem, but forget to write here. It doesn’t matter, how you start Ray Serve (via CLI or Python API), you can specify a parameters --http-host and --http-port. This will open the server for external requests.

CLI:
serve start --http-host "xxx.xx.xxx.xx" --http-port xxxx
Host is string type, port is integer type.

Python API:

from ray import serve

serve.start(
    http_options= {
        "host": 'xxx.xx.xxx.xx',
        "port": xxxx
    }

Also you can specify location parameter, that deploy your application on the head node, or on all nodes that your cluster have.

Thank you for attention!

5

@GolikovAndrey Excellent. You could also use a proxy for auth if you wanted and then have it routte to the appropriate Ray Serve deployment.

Should we consider this resolved then?

Happy to help (HTH)

1 Like
6

Yes, we can consider this problem solved.

7

excellent, and hope all works for you