# Batching doesn't work: requests are processed one by one

**URL:** <https://discuss.ray.io/t/batching-doesnt-work-requests-are-processed-one-by-one/2508>\
**Category:** Ray Serve\
**Created:** [June 14, 2021, 1:22am UTC](https://discuss.ray.io/t/batching-doesnt-work-requests-are-processed-one-by-one/2508 "2021-06-14T01:22:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![blshao84](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blshao84/32/1010_2.png) [@blshao84](https://discuss.ray.io/u/blshao84)\
**Post date:** [June 14, 2021, 1:22am UTC](https://discuss.ray.io/t/batching-doesnt-work-requests-are-processed-one-by-one/2508/1 "2021-06-14T01:22:08Z")

</div>

Hi, I’m experimenting serve.batch:

First of all, I start a local cluster by `ray start --head`. Then I launch serve by ‘serve start’. Next, I deploy below simple code to the serve:  
`  
import time  
from typing import List  
from pydantic import BaseModel  
from ray import serve  
from starlette.requests import Request

class PricingRequest(BaseModel):  
valuation\_date: str  
pid: str  
payoff: dict  
cmm: dict

@serve.deployment(’/present\_value’)  
class PricingApi:  
@serve.batch  
async def present\_value(self, reqs: List[Request]):  
print(f’batched {len(reqs)} reqs’)  
data = []  
for r in reqs:  
p = await r.json()  
data.append(p)  
return [{‘pid’: r[‘pid’], ‘pv’: 0.0} for r in data]

```
async def __call__ (self, req: Request):
    return await self.present_value(req)

```

ray.init(namespace=‘serve’)  
PricingApi.deploy()  
`  
So far, everything works fine and I’m able to call the api by curl post. Then I start a Java application that sends post requests in parallel. However, from the server, I saw all logs like ‘batched 1 reqs’. I understand serve.batch is opportunistic but I would assume above example should work.

I’m using latest nightly build and python 3.8.

Thanks,  
-BS

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [June 17, 2021, 4:49am UTC](https://discuss.ray.io/t/batching-doesnt-work-requests-are-processed-one-by-one/2508/2 "2021-06-17T04:49:52Z")

</div>

Hi! If your request returns instantly there might not be time to accumulate a batch of requests. Does it behave as expected if you add a `time.sleep` in the request to simulate a time-intensive computation? Alternatively, you could set `@serve.batch(batch_wait_timeout_s=10)` which would wait 10 seconds to accumulate a batch before processing. See also [Batching Tutorial — Ray v2.0.0.dev0](https://docs.ray.io/en/master/serve/tutorials/batch.html)

---

<div class="post-metadata">

**Author:** ![blshao84](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blshao84/32/1010_2.png) [@blshao84](https://discuss.ray.io/u/blshao84)\
**Post date:** [June 19, 2021, 5:37am UTC](https://discuss.ray.io/t/batching-doesnt-work-requests-are-processed-one-by-one/2508/3 "2021-06-19T05:37:36Z")

</div>

Thanks Archit, it works. I will continue my prototyping 🙂