# Low latency runtime inference **URL:** \ **Category:** Ray Serve\ **Created:** [April 9, 2025, 11:51am UTC](https://discuss.ray.io/t/low-latency-runtime-inference/22266 "2025-04-09T11:51:41Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![mbogomilov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@mbogomilov](https://discuss.ray.io/u/mbogomilov)\ **Post date:** [April 9, 2025, 11:51am UTC](https://discuss.ray.io/t/low-latency-runtime-inference/22266/1 "2025-04-09T11:51:41Z")
**1. Severity of the issue: (select one)** None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. [x] High: Completely blocks me. **2. Environment:** - Ray version: 2.44.1 - Python version: 3.10.12 - OS: Ubuntu 22.04.1 LTS - Cloud/Infrastructure: on premise - Other libs/tools (if relevant): CatBoost regression, gRPC **3. What happened vs. what you expected:** - Expected: gRPC runtime inference latency \< 5ms in single threaded mode - Actual: latency \> 10 ms, whether client is run on same machine or another # serve\_model.py import ray from ray import serve import pandas as pd from catboost import CatBoostRegressor import time import datetime import inference\_pb2 import inference\_pb2\_grpc from fastapi import Request #@serve.deployment(num\_replicas=8, ray\_actor\_options={“num\_cpus”: 2.0, “num\_gpus”: 0}) @serve.deployment class CatBoostModel: def **init** (self): self.model = CatBoostRegressor() self.model.load\_model(“catboost\_model.cbm”) ``` async def __call__ (self, request: Request): start_time = datetime.datetime.now() body = await request.json() #features = pd.DataFrame([body["features"]], columns=["f1"]) features = pd.DataFrame([body["features"]], columns=["f1", "f2", "f3"]) pred = self.model.predict(features)[0] resp = {"prediction": pred} end_time = datetime.datetime.now() dur = int((end_time - start_time).microseconds/1000) print(f'REST inference: {dur}') return resp async def Predict(self, request): start_time = datetime.datetime.now() #print(f'request.features class: {request.features. __class__ }') features = list(request.features) #print(f'features: {features}') df = pd.DataFrame([features], columns=["f1", "f2", "f3"]) #print(f'dataframe shape: {df.shape}') #print(f'dataframe info: {df.info()}') #print(f'dataframe: {df}') pred = self.model.predict(df)[0] resp = inference_pb2.PredictResponse(result=pred) end_time = datetime.datetime.now() dur = int((end_time - start_time).microseconds/1000) print(f'gRPC inference: {dur}') return resp ``` g = CatBoostModel.bind() app1 = “app1” serve.run(target=g, name=app1, route\_prefix=f"/{app1}") # inference.proto syntax = “proto3”; package inference; service InferenceService { rpc Predict (PredictRequest) returns (PredictResponse); } message PredictRequest { repeated float features = 1; } message PredictResponse { float result = 1; } # generate proto python -m grpc\_tools.protoc -I. --python\_out=. --grpc\_python\_out=. inference.proto # start serve serve start --grpc-port 9000 --grpc-servicer-functions inference\_pb2\_grpc.add\_InferenceServiceServicer\_to\_server python serve\_model.py # client # grpc\_client.py import grpc import inference\_pb2 import inference\_pb2\_grpc import time import threading import datetime NUM\_THREADS = 1 REQUESTS\_PER\_THREAD = 1000 #NUM\_THREADS = 8 #REQUESTS\_PER\_THREAD = 125 channel = grpc.insecure\_channel(“localhost:9000”) stub = inference\_pb2\_grpc.InferenceServiceStub(channel) features = [0.5, 1.2, -0.7] def worker(thread\_id): for i in range(REQUESTS\_PER\_THREAD): start\_time = datetime.datetime.now() #start = time.perf\_counter() #response = stub.Predict(inference\_pb2.PredictRequest(features=features), compression=grpc.Compression.Gzip) response = stub.Predict(inference\_pb2.PredictRequest(features=features)) end\_time = datetime.datetime.now() dur = int((end\_time - start\_time).microseconds/1000) #end = time.perf\_counter() #latency = (end - start) \* 1000 print(f"[Thread {thread\_id}] Response: {response.result} | Latency: {dur:.2f} ms") ``` #print("gRPC prediction:", res.result) #print(f"gRPC latency: {(end - start) * 1000:.2f} ms") ``` # Launch threads threads = for t\_id in range(NUM\_THREADS): t = threading.Thread(target=worker, args=(t\_id,)) threads.append(t) t.start() # Wait for all threads to complete for t in threads: t.join() print(“Load test completed.”) # bash python grpc\_client\_multi.py |grep Latency |awk ‘{s+=$7; c++} END {print s/c}’ |bc -l 10.113 ms ---
**Author:** ![abrarsheikh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/abrarsheikh/32/8019_2.png) [@abrarsheikh](https://discuss.ray.io/u/abrarsheikh)\ **Post date:** [April 14, 2025, 4:41pm UTC](https://discuss.ray.io/t/low-latency-runtime-inference/22266/2 "2025-04-14T16:41:55Z")
Thank you for reporting this issue. Just so that it helps me narrow down the problem, have you only started noticing this with 2.44.1 and the latency with older version was \< 5ms ---
**Author:** ![mbogomilov](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@mbogomilov](https://discuss.ray.io/u/mbogomilov)\ **Post date:** [April 16, 2025, 12:19am UTC](https://discuss.ray.io/t/low-latency-runtime-inference/22266/3 "2025-04-16T00:19:51Z")
Hi, I have another setup with CentOS 7 and Ray 2.4.0 and there it is 2x faster. ---
**Author:** ![Gene](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gene/32/4701_2.png) [@Gene](https://discuss.ray.io/u/Gene)\ **Post date:** [April 16, 2025, 5:32pm UTC](https://discuss.ray.io/t/low-latency-runtime-inference/22266/4 "2025-04-16T17:32:33Z")
@mbogomilov we only started offering gRPC proxy since Ray 2.7.0. Are you comparing the performance of Serve’s HTTP proxy at Ray 2.4.0 to gRPC proxy at Ray 2.44.1 or are you actually on a different version of Ray as you previously stated?