# Ray Serve Model Worker Replicas Created But GPU Usage is 0% during Inference **URL:** \ **Category:** Ray Serve\ **Created:** [January 11, 2022, 8:43pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662 "2022-01-11T20:43:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![shamil](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@shamil](https://discuss.ray.io/u/shamil)\ **Post date:** [January 11, 2022, 8:43pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/1 "2022-01-11T20:43:08Z")
Hi All, Up until yesterday, I was able to see Volatile GPU-Util spike up during inference on a model utilizing the 3 replicas I made. However, for no real reason, all a sudden the GPU-Util never changes from 0% even during inference. I am unsure what has changed. I verified and everything looks good on code part. Any idea? ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/c/c484445f1768a5684864cfddfa996ff5aa3810af.png) ---
**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\ **Post date:** [January 11, 2022, 10:01pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/2 "2022-01-11T22:01:06Z")
Volatile GPU Util is instantaneous and mostly an estimate, see [cuda - nvidia-smi Volatile GPU-Utilization explanation? - Stack Overflow](https://stackoverflow.com/questions/40937894/nvidia-smi-volatile-gpu-utilization-explanation). Did the inference latency spike up as well? ---
**Author:** ![shamil](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@shamil](https://discuss.ray.io/u/shamil)\ **Post date:** [January 11, 2022, 10:08pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/3 "2022-01-11T22:08:32Z")
Hi @simon-mo , I had same thoughts. The answer is no, the inference latency time did not spike up. It in fact remained the same! This is making me question if I even need gpu? When I make 3 model replicas, by default they go into 3 cores and I can see their usage spike. The question is can a model replica actor only require gpu, and not cpu? I feel cpu computation is being taken care by cpu, so the gpu actors are completely ignored. ---
**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\ **Post date:** [January 11, 2022, 10:58pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/4 "2022-01-11T22:58:18Z")
Unfortunately this entire depends on your workload I would recommend microbenchmark your model on cpu vs gpu to compare their latency ---
**Author:** ![shamil](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@shamil](https://discuss.ray.io/u/shamil)\ **Post date:** [January 11, 2022, 11:06pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/5 "2022-01-11T23:06:09Z")
Correct, I figured i may not need gpu in my case. I forced ray serve to not use cpu by ray\_actor\_options={“num\_cpus”: 0,“num\_gpus”: 0.33},num\_replicas = 3 However, dashboard still shows CPU usage spiking up whereas GPU usage is minimal. Would be nice to know why that may be happening. Expected behaviour would be to only see gpu usage. ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/a/a1edd469e7f5cff12a74e7fdb7546e152a499512.png) ---
**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\ **Post date:** [January 11, 2022, 11:40pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/6 "2022-01-11T23:40:57Z")
“num\_cpus”: 0 is only just placement constraint, not hard utilization constraint. Your application will use CPU regardless of this value. Ray doesn’t automatically transform your model to use CPU vs GPU. You should take a look at the model to choose what to run on CPU vs GPU. Even if the entire model is on GPU, there are cost of coordinating the GPU instructions from CPU and movement. ---
**Author:** ![shamil](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@shamil](https://discuss.ray.io/u/shamil)\ **Post date:** [January 17, 2022, 9:58pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/7 "2022-01-17T21:58:58Z")
@simon-mo Thanks, I am in agreement with costs of coordinating GPU from CPU. I did an experiment to run same model one using Ray’s API vs Ray’s remote. The code uses GPU only when I use Ray’s remote but not when I create model as API endpoint. Here is the code: Ray Serve Model API: @serve.deployment(route\_prefix="/sentiment",ray\_actor\_options={“num\_gpus”: 0.5},num\_replicas = 2) class MLflowBackend: def **init** (self, model\_uri): spacy.prefer\_gpu() self.model = mlflow.pyfunc.load\_model(model\_uri=model\_uri) ``` async def __call__ (self, request): data = await request.body() df = pd.DataFrame([data]) cats = self.model.predict(df) return cats['predictions'][0] ``` #model inference for i in data[:6000]: response = requests.post(“[http://127.0.0.1:8000/sentiment](http://127.0.0.1:8000/sentiment)”,data = i) Ray’ Serve Using Remote: @serve.deployment(name=“sentiment”,ray\_actor\_options={“num\_gpus”: 0.5},num\_replicas = 2) class MLflowBackend1: def **init** (self, model\_uri): spacy.prefer\_gpu() self.model = mlflow.pyfunc.load\_model(model\_uri=model\_uri) ``` def __call__ (self, request): data = request df = pd.DataFrame([data]) cats = self.model.predict(df) return cats['predictions'][0] ``` sentiment = MLflowBackend1.get\_handle() #model inference for i in data[:10000]: r= sentiment .remote(text) ret = ray.get(r) Overall, slight change but one uses GPU while other method using endpoint does not. Any particular reason why? ---
**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\ **Post date:** [January 19, 2022, 6:55pm UTC](https://discuss.ray.io/t/ray-serve-model-worker-replicas-created-but-gpu-usage-is-0-during-inference/4662/8 "2022-01-19T18:55:11Z")
Both should use GPU. Your second case (using handle.remote) shows GPU usage probably because it is fast to directly send it instead of parsing HTTP request. Since the second case has higher load, the GPU usage actually shows up. The first case probably still use GPU but because the load is smaller (less throughput due to the HTTP parsing overhead), the GPU is not saturated and usage is not showing up. You can use a load test tool (for example locust) to increase the http load to visualize GPU usage.