# Understanding Ray Serve Memory Consumption **URL:** \ **Category:** Ray Serve\ **Created:** [December 14, 2023, 4:19pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162 "2023-12-14T16:19:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![tzhang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tzhang/32/4545_2.png) [@tzhang](https://discuss.ray.io/u/tzhang)\ **Post date:** [December 14, 2023, 4:19pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/1 "2023-12-14T16:19:35Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. I’m hosting a ray serve (single node) application for a ML inference API service, and getting OOM error. I’m running it on a GCP machine with 4 CPUs, 16GB CPU memory, 1 GPU with 15GB memory (n1-standard-4). The model is a distilbert model (~500mb) and it’s loaded on GPU. I run with 10 replicas and when I send some requests I get OOM error on CPU (\> 0.95 threshold), causing the resource manager to kill actors and resulting in 500 internal server error on the killed request. And when I check the CPU consumption with `top`, I see: ```auto PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 292 xxxxxxxx 35 15 16.2g 2.0g 580528 S 0.7 7.4 8:27.45 ray::ServeRepli 293 xxxxxxxx 35 15 16.2g 2.1g 582464 S 0.7 7.4 8:27.42 ray::ServeRepli 294 xxxxxxxx 35 15 16.2g 2.0g 583848 S 0.7 7.4 8:27.28 ray::ServeRepli 295 xxxxxxxx 35 15 16.1g 2.0g 584568 S 0.7 7.4 8:28.87 ray::ServeRepli 314 xxxxxxxx 35 15 16.2g 2.0g 584668 S 0.7 7.3 8:27.19 ray::ServeRepli 338 xxxxxxxx 35 15 16.1g 2.0g 584172 S 0.7 7.3 8:27.38 ray::ServeRepli ``` If the model itself is on GPU, why does each of the replica taking so much CPU memory, and is there a way to reduce the CPU usage? My deployment config: ```auto - name: MLAPI num_replicas: 10 ray_actor_options: num_cpus: 0.3 num_gpus: 0.1 ``` Ray status resource usage: ```auto Usage: 3.0/4.0 CPU 1.0/1.0 GPU 0B/9.47GiB memory 44B/4.74GiB object_store_memory ``` Any help would be appreciated! ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [December 14, 2023, 5:28pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/2 "2023-12-14T17:28:23Z")
Changing the category to Ray Serve. cc: @Gene for thoughts ---
**Author:** ![Gene](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gene/32/4701_2.png) [@Gene](https://discuss.ray.io/u/Gene)\ **Post date:** [December 14, 2023, 5:40pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/3 "2023-12-14T17:40:05Z")
Not an expert on this, but I feel this has to do with how the model is loaded. Have you tried using `model.to("cuda:0")` to send the model to the GPU? This article might help debug where the memory is leaked [Memory Leakage with PyTorch. If you’re reading this post, then most… | by Raghad Alghonaim | Medium](https://medium.com/@raghadalghonaim/memory-leakage-with-pytorch-23f15203faa4) ---
**Author:** ![Gene](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gene/32/4701_2.png) [@Gene](https://discuss.ray.io/u/Gene)\ **Post date:** [December 14, 2023, 5:47pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/4 "2023-12-14T17:47:13Z")
Also just want to highlight those `ray_actor_options` are “logical” resources [Resource Allocation — Ray 2.8.1](https://docs.ray.io/en/latest/serve/resource-allocation.html) Ray Serve doesn’t prevent users to specify the resources more than what’s physically presented on the machine. Users would need to measure how much each type of resources their application needs and use the options such as `num_cpus`, `num_gpus`, `memory`…etc to tell Serve not to over schedule replicas if the cluster doesn’t have enough resources. ---
**Author:** ![tzhang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tzhang/32/4545_2.png) [@tzhang](https://discuss.ray.io/u/tzhang)\ **Post date:** [December 14, 2023, 6:46pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/5 "2023-12-14T18:46:01Z")
I’m pretty sure the model is loaded on GPU since `nvidia-smi` command shows GPU memory usage correctly for the number of replicas I have created. ---
**Author:** ![tzhang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tzhang/32/4545_2.png) [@tzhang](https://discuss.ray.io/u/tzhang)\ **Post date:** [December 14, 2023, 6:47pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/6 "2023-12-14T18:47:42Z")
Thanks for the pointer. Yes, I read about it and I think I understand that. I guess my question was really what is exactly happening with that 7.4% memory per replica, from my other logs, that’s basically ~1.6GB of memory taken from CPU. ---
**Author:** ![Gene](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gene/32/4701_2.png) [@Gene](https://discuss.ray.io/u/Gene)\ **Post date:** [December 14, 2023, 11:07pm UTC](https://discuss.ray.io/t/understanding-ray-serve-memory-consumption/13162/7 "2023-12-14T23:07:27Z")
My best guess is maybe there are some references to the model in your deployment code? Or issue with pytorch? Maybe this thread helps? [How to free CPU RAM after `module.to(cuda_device)`? - PyTorch Forums](https://discuss.pytorch.org/t/how-to-free-cpu-ram-after-module-to-cuda-device/20381) Can setup [memory\_profiler](https://pypi.org/project/memory_profiler/) to track the memory usage.