# Model replication with multiple GPU deployments

**URL:** <https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475>\
**Category:** Ray Serve\
**Created:** [December 17, 2021, 8:05pm UTC](https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475 "2021-12-17T20:05:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![puntime\_error](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/puntime_error/32/502_2.png) [@puntime\_error](https://discuss.ray.io/u/puntime_error)\
**Post date:** [December 17, 2021, 8:05pm UTC](https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475/1 "2021-12-17T20:05:00Z")

</div>

I have 2 CUDA GPU resources on my server that is running ray. If I deploy a single model replica with `num_gpus": 1}`, things are fine. If I set the number of replicas to 2, I get a "RuntimeError: CUDA out of memory. " error when Ray tries to deploy the 2nd replica. How are people deploying multiple replicas with multiple GPUs? Any sample code/gist or suggestions to try would be appreciated.

---

<div class="post-metadata">

**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\
**Post date:** [December 20, 2021, 7:27pm UTC](https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475/2 "2021-12-20T19:27:41Z")

</div>

Hi @puntime_error, when you set the replicas to 2 and keep num\_gpus to 1, there should be two identical process created, each taking one GPU with their corresponding CUDA\_VISIBLE\_DEVICES.

Can you take a look at `nvidia-smi` before and after and make sure there are no other processes running on the two GPUs?

---

<div class="post-metadata">

**Author:** ![Sujit\_Kumar](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sujit_kumar/32/3018_2.png) [@Sujit\_Kumar](https://discuss.ray.io/u/Sujit_Kumar)\
**Post date:** [August 16, 2022, 1:09pm UTC](https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475/3 "2022-08-16T13:09:25Z")

</div>

@simon-mo @puntime_error am unable to use multiple gpus while serving ray.  
I have raised an issue here [Issue on page /serve/getting\_started.html · Issue #27905 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/27905)  
Can you please help?

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [August 16, 2022, 6:15pm UTC](https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475/4 "2022-08-16T18:15:26Z")

</div>

Hi @Sujit_Kumar, to narrow down the issue, do the basic GPU examples at [GPU Support — Ray 3.0.0.dev0](https://docs.ray.io/en/master/ray-core/tasks/using-ray-with-gpus.html) work for you? You can try it on Ray 2.0.0rc1 (`pip install "ray[serve, default]==2.0.0rc1"`)

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [August 16, 2022, 6:24pm UTC](https://discuss.ray.io/t/model-replication-with-multiple-gpu-deployments/4475/5 "2022-08-16T18:24:59Z")

</div>

@Sujit_Kumar you may also have to tell the `transformers` library to use GPU; see [How to make transformers examples use GPU? · Issue #2704 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/2704#issuecomment-1016971649) for example and the surrounding context.