Using asyncio to process HTTP requests concurrently

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

Hello guys, as the following sample code shows, I make my servable methods async def for the purpose of processing HTTP requests concurrently.
The expected output is “one one one one one two two two two two", however, the actual output is “one two one two one two one two one two” .
It seems that the requests are not executed asynchronously but serially. Where is my mistake？

from random import random
import requests
import ray
from ray import serve
import time
import asyncio

@serve.deployment(num_replicas=10, route_prefix="/composed")
class ComposedModel:
    def __init__(self):
        print("hello, world!")

    # This method can be called concurrently!
    async def __call__(self, starlette_request):
        print("One")
    
        await asyncio.sleep(5)
        
        print("Two")

        return 0

ray.init()
serve.start()
ComposedModel.deploy()

for _ in range(5):
    resp = requests.get("http://127.0.0.1:8000/composed", data="hey!")

THANKS!!!

2

Hi @liangzhimin, your serve deployment is completely correct! However, you’re sending the requests in sequence, not in parallel.

Thus, the second request is only sent after the first one finished, etc.

You can easily leverage Ray to send requests in parallel. Replace your last two lines by this:

@ray.remote(num_cpus=0)
def send_request():
    return requests.get("http://127.0.0.1:8000/composed", data="hey!")

fut = [send_request.remote() for _ in range(5)]
ray.get(fut)

Output:

2021-08-03 14:59:02,261	INFO services.py:1256 -- View the Ray dashboard at http://127.0.0.1:8265
(pid=8635) 2021-08-03 14:59:04,883	INFO http_state.py:75 -- Starting HTTP proxy with name 'QzIcuW:SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-node:10.245.45.174-0' on node 'node:10.245.45.174-0' listening on '127.0.0.1:8000'
2021-08-03 14:59:05,022	INFO api.py:713 -- Started Serve instance in namespace '33c2fe1a-1295-4b77-a325-18e749619f2d'.
2021-08-03 14:59:05,028	INFO api.py:414 -- Updating deployment 'ComposedModel'.
(pid=8645) INFO:     Started server process [8645]
(pid=8635) 2021-08-03 14:59:05,099	INFO backend_state.py:869 -- Adding 10 replicas to backend 'ComposedModel'.
(pid=8643) hello, world!
(pid=8646) hello, world!
(pid=8640) hello, world!
(pid=8638) hello, world!
(pid=8631) hello, world!
(pid=8642) hello, world!
(pid=8639) hello, world!
(pid=8637) hello, world!
(pid=8632) hello, world!
(pid=8634) hello, world!
(pid=8643) One
(pid=8640) One
(pid=8632) One
(pid=8634) One
(pid=8638) One
(pid=8643) Two
(pid=8640) Two
(pid=8632) Two
(pid=8634) Two
(pid=8638) Two
(pid=8635) 2021-08-03 14:59:11,785	INFO backend_state.py:880 -- Removing 10 replicas from backend 'ComposedModel'.
3 Likes
3

Thanks!!! You help me a lot!!!