# Ray serve autoscaling queue size

**URL:** <https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219>\
**Category:** Ray Serve\
**Created:** [May 22, 2022, 5:30pm UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219 "2022-05-22T17:30:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![plum9](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@plum9](https://discuss.ray.io/u/plum9)\
**Post date:** [May 22, 2022, 5:30pm UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219/1 "2022-05-22T17:30:58Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi,  
I’ve deployed a Multi-modal ML model using Ray serve with autoscaling config. The autoscaling policies have max\_concurrent\_queries set to 1000. From docs, it looks like it’s indicative of queue size per replica which stores requests which are yet to processed or have given a response. I want to use this deployment for a large-scale inference where I can submit anywhere from 5000, to 1,000,000 requests in an async fashion. Currently, when I try to submit 5000 requests to the deployment (which is scaled up to 5 replicas with 1000 max\_concurrent\_queries) it stops accepting requests around 2000.

The head node is p3.2xlarge with 1 GPU and the scaling policies required 2 more GPU’s so it scales up. The ray\_actor\_options are num\_cpus:2, num\_gpus:0.5. I’m using the latest ray version 1.12.1

How does the ServeHandle distribute the requests among replicas?  
Shouldn’t all the requests be equally divided among 5 replica queues?  
Is there a limitation to max\_concurrent\_queries?  
Is my understanding of max\_concurrent\_queries correct where it’s synonymous to queue per replica?

---

<div class="post-metadata">

**Author:** ![Sihan\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sihan_wang/32/2570_2.png) [@Sihan\_Wang](https://discuss.ray.io/u/Sihan_Wang)\
**Post date:** [May 23, 2022, 8:46pm UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219/2 "2022-05-23T20:46:37Z")

</div>

Hi,

handle will use round robin to dispatch the request to each replica, when the number of ongoing queries of one replica is greater than max\_concurrent\_queries. It will stop assigning more queries to the replica and try to assign the query to other one until the “busy” replica’s ongoing queries is smaller than max\_concurrent\_queries.  
And max\_concurrent\_queries is equal to number of running queries on replica plus the number of queuing queries on replica

---

<div class="post-metadata">

**Author:** ![plum9](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@plum9](https://discuss.ray.io/u/plum9)\
**Post date:** [May 23, 2022, 9:03pm UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219/3 "2022-05-23T21:03:24Z")

</div>

@Sihan_Wang  
So in the above case, where max\_concurrent\_queries is set to 1000 and I’ve 5 replicas, by extension of round-robin, it should queue up all the 5000 requests, right? One request takes roughly 10 seconds to process. I’m confused as to why it stops at about 2000.

Here is a snippet from config

```auto
ray_actor_options={"num_cpus": 1, "num_gpus": 0.5},
                _autoscaling_config={
                    "min_replicas": 5,
                    "max_replicas": 5,
                    "target_num_ongoing_requests_per_replica": 100,
                },
                version="v1",
                max_concurrent_queries=1000
            )

```

Noticed the same behavior while min\_replicas was set to 2

---

<div class="post-metadata">

**Author:** ![Sihan\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sihan_wang/32/2570_2.png) [@Sihan\_Wang](https://discuss.ray.io/u/Sihan_Wang)\
**Post date:** [May 23, 2022, 10:09pm UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219/4 "2022-05-23T22:09:56Z")

</div>

Hi @plum9 ,

With current information, it is difficult for me to root cause the issue.  
Could you provide a simple script that I can reproduce the issue on my end? (either cpu or gpu env is good to me as long as i can reproduce the issue)

Thanks  
Sihan

---

<div class="post-metadata">

**Author:** ![plum9](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@plum9](https://discuss.ray.io/u/plum9)\
**Post date:** [May 24, 2022, 12:50am UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219/5 "2022-05-24T00:50:39Z")

</div>

Thanks @Sihan_Wang I’ll create a short script to replicate.  
In meantime, is there a limitation on max\_concurrent\_queries in terms of upper limit?

---

<div class="post-metadata">

**Author:** ![plum9](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@plum9](https://discuss.ray.io/u/plum9)\
**Post date:** [May 24, 2022, 8:12pm UTC](https://discuss.ray.io/t/ray-serve-autoscaling-queue-size/6219/6 "2022-05-24T20:12:08Z")

</div>

I made a mistake while coming up with autoscaling policies. Closing this. Thanks Sihan!