# Ray serve blocking requests when serving an LLM

**URL:** <https://discuss.ray.io/t/ray-serve-blocking-requests-when-serving-an-llm/15981>\
**Category:** Ray Serve\
**Created:** [October 10, 2024, 12:25am UTC](https://discuss.ray.io/t/ray-serve-blocking-requests-when-serving-an-llm/15981 "2024-10-10T00:25:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Steve\_Li](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/steve_li/32/6568_2.png) [@Steve\_Li](https://discuss.ray.io/u/Steve_Li)\
**Post date:** [October 10, 2024, 12:25am UTC](https://discuss.ray.io/t/ray-serve-blocking-requests-when-serving-an-llm/15981/1 "2024-10-10T00:25:09Z")

</div>

Hey folks, working on serving an llm model using `ray serve` with FastAPI. I’ve implemented batching of requests on my endpoint, but I’ve noticed that each batch happens serially instead of concurrently.

My guess is that this is a limitation of the asyncio event loop - calling inference on the model is a blocking operation, and the GIL prevents multiple threads happening in true parallel fashion. Hence, we have a single event loop that waits for a batch to complete before waiting for another batch.

I’ve tried utilizing threading with FastAPI by removing any `async` or `await` in my code, but it seems that calling `remote()` on a deployment creates future objects, `DeploymentResponse`, that need to be awaited.

For my use case, I want to be able to serve as many requests as possible while maintaining efficient batching. That is, simply increasing batch size will reduce throughput due to latency in calculating the forward pass. Is there a way to do so with the current `ray serve` setup?

I’m not too concerned with OOM issues at the moment - I simply want to call the gpu as many times as possible in parallel.

---

<div class="post-metadata">

**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\
**Post date:** [October 15, 2024, 8:52pm UTC](https://discuss.ray.io/t/ray-serve-blocking-requests-when-serving-an-llm/15981/2 "2024-10-15T20:52:55Z")

</div>

Hi @Steve_Li do you happen to be using vLLM? If that’s the case, you can use the async engine interface which will play nicely with asyncio: [AsyncLLMEngine — vLLM](https://docs.vllm.ai/en/latest/dev/engine/async_llm_engine.html).

Otherwise, I would suggest wrapping the synchronous model call with [run\_in\_executor](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor) to run it on a thread pool and avoid blocking the asyncio loop (make sure your model is thread safe or use an executor pool of size 1).

---

<div class="post-metadata">

**Author:** ![Steve\_Li](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/steve_li/32/6568_2.png) [@Steve\_Li](https://discuss.ray.io/u/Steve_Li)\
**Post date:** [October 20, 2024, 8:59pm UTC](https://discuss.ray.io/t/ray-serve-blocking-requests-when-serving-an-llm/15981/3 "2024-10-20T20:59:05Z")

</div>

Hmmmm I did try `run_in_executor`, but I still ran into the same concurrency issues. Seems that the actual llm model doesn’t play nicely with the GIL.

I’ll try using vLLM, I also was looking into their Async engine.

---

<div class="post-metadata">

**Author:** ![Gene](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gene/32/4701_2.png) [@Gene](https://discuss.ray.io/u/Gene)\
**Post date:** [October 20, 2024, 9:03pm UTC](https://discuss.ray.io/t/ray-serve-blocking-requests-when-serving-an-llm/15981/4 "2024-10-20T21:03:39Z")

</div>

Not specifically about batching, but there’re some example code using Ray Serve to serve LLM with vLLM at [Serve a Large Language Model with vLLM — Ray 2.37.0](https://docs.ray.io/en/latest/serve/tutorials/vllm-example.html)