I’d like to know if there’s a way to “bucket” batch requests on serve. Something along the lines of the following snippet:
@serve.batch(bucket_by="batch_param") async def batch_process(self, samples: list[np.ndarray], batch_param: float): batch = build_batch(samples) return process_batch(batch, batch_param) async def process(self, sample: np.ndarray, param: float): await batch_process(sample, param)
In which batch queues would be bucketed by
param, i.e. requests with different
param values would be queued in different batches.