After creating endpoints, what is the elegant way to keep Ray serve alive so that clients can request at any time?
For now, I use “while True” to keep scirpt from quiting.
After creating endpoints, what is the elegant way to keep Ray serve alive so that clients can request at any time?
For now, I use “while True” to keep scirpt from quiting.
Hi @matrixyy, the recommended way is to create a long-lived Ray instance in the background and deploy Serve to it:
# Start Ray and Serve in the background.
ray start --head
serve start --http-host=0.0.0.0 # If you want to expose Serve on the network.
With this mode of deployment, Serve will keep running in the background until you explicitly call ray stop
.
Hi @eoakes The Ray version is 1.2. Are you sure your answer works for 1.2? The following code works (my previous method):
import ray
from ray import serve
import requests
ray.init(num_cpus=4)
client = serve.start()
def say_hello(request):
return "hello " + request.query_params["name"] + "!"
# Form a backend from our function and connect it to an endpoint.
client.create_backend("my_backend", say_hello)
client.create_endpoint("my_endpoint", backend="my_backend", route="/hello")
count = 1
while True:
count += 1
if (count % 1000 == 0):
print('loop 1000, sleep 1s')
time.sleep(1)
However, when I use your method (remove while loop, and execute the two command first, the command can execute successfully), there is error when sending http request:
Path /hello not found. Please ping http://.../-/routes for routing table
For version 1.2, I can’t directly use
serve.create_backend
What only I can use is:
client = serve.start()
client.create_backend
It seems in your method, there is no need to call serve.start() in the code.
Hi @matrixyy,
Assuming you are on Ray 1.2.0, after running @eoakes 's commands to start a long-running Ray cluster and Serve instance, you will need to run the following code in Python:
ray.init(address="auto") # Connect to the running Ray cluster.
client = serve.connect() # Connect to the running Serve instance.
Then you can use the client
as usual in your code.
In general you’ll want to make sure you’re using the right version of the documentation. Here’s the documentation for the nightly build: Ray Serve: Scalable and Programmable Serving — Ray v2.0.0.dev0
And here’s the documentation for the latest pip release (1.2.0): Ray Serve: Scalable and Programmable Serving — Ray v1.2.0
Hi @architkulkarni It works! (use connect, not start) Thanks! But there is another question. What is the correct way to include a file? My service API is simple: just load a cvs and return the IP based on the query parameter:
def service_return(request):
df_csv = pd.read_csv('../../data/final_result.csv')
## ouput = some opertions on df_csv
return output
When I first use your method, there is an error in the log:
../../data/final_result.csv
But when I re-start all service and command again. There is no error and it can return the result successfully. Very weird!
No difference, you can use either one depending on whether you prefer the command-line interface or Python. The command serve start
actually just calls serve.start(detached=True)
in Python.