Using asyncio to process HTTP requests concurrently

Hello guys, as the following sample code shows, I make my servable methods async def for the purpose of processing HTTP requests concurrently.
The expected output is “one one one one one two two two two two", however, the actual output is “one two one two one two one two one two” .
It seems that the requests are not executed asynchronously but serially. Where is my mistake?

from random import random
import requests
import ray
from ray import serve
import time
import asyncio

@serve.deployment(num_replicas=10, route_prefix="/composed")
class ComposedModel:
    def __init__(self):
        print("hello, world!")

    # This method can be called concurrently!
    async def __call__(self, starlette_request):
        await asyncio.sleep(5)

        return 0


for _ in range(5):
    resp = requests.get("", data="hey!")


Hi @liangzhimin, your serve deployment is completely correct! However, you’re sending the requests in sequence, not in parallel.

Thus, the second request is only sent after the first one finished, etc.

You can easily leverage Ray to send requests in parallel. Replace your last two lines by this:

def send_request():
    return requests.get("", data="hey!")

fut = [send_request.remote() for _ in range(5)]


2021-08-03 14:59:02,261	INFO -- View the Ray dashboard at
(pid=8635) 2021-08-03 14:59:04,883	INFO -- Starting HTTP proxy with name 'QzIcuW:SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-node:' on node 'node:' listening on ''
2021-08-03 14:59:05,022	INFO -- Started Serve instance in namespace '33c2fe1a-1295-4b77-a325-18e749619f2d'.
2021-08-03 14:59:05,028	INFO -- Updating deployment 'ComposedModel'.
(pid=8645) INFO:     Started server process [8645]
(pid=8635) 2021-08-03 14:59:05,099	INFO -- Adding 10 replicas to backend 'ComposedModel'.
(pid=8643) hello, world!
(pid=8646) hello, world!
(pid=8640) hello, world!
(pid=8638) hello, world!
(pid=8631) hello, world!
(pid=8642) hello, world!
(pid=8639) hello, world!
(pid=8637) hello, world!
(pid=8632) hello, world!
(pid=8634) hello, world!
(pid=8643) One
(pid=8640) One
(pid=8632) One
(pid=8634) One
(pid=8638) One
(pid=8643) Two
(pid=8640) Two
(pid=8632) Two
(pid=8634) Two
(pid=8638) Two
(pid=8635) 2021-08-03 14:59:11,785	INFO -- Removing 10 replicas from backend 'ComposedModel'.

Thanks!!! You help me a lot!!!