# Not sure how num\_replicas works

**URL:** <https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057>\
**Category:** Ray Serve\
**Created:** [March 1, 2021, 2:22pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057 "2021-03-01T14:22:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tim](https://avatars.discourse-cdn.com/v4/letter/t/ce73a5/32.png) [@tim](https://discuss.ray.io/u/tim)\
**Post date:** [March 1, 2021, 2:22pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057/1 "2021-03-01T14:22:29Z")

</div>

Hello there,

I am serving pytorch models using ray serve. I have got 2 GPUs and I want to serve only one model for the moment.

I tried scaling out my model to 2 replicas, using

```auto
config = {"num_replicas": 2}
client.create_backend("resnet18:v0", ImageModel, ray_actor_options={"num_cpus": 6, "num_gpus": 1}, config=config)

```

This works. When I look at nvidia-smi however, I can see that each GPU is used only at 12% maximum, each. So I was wondering: why not set num\_replicas to 4 and set “num\_gpus” to 0.5? But unfortunately, it does not work.

Can you explain me why ? (Note: I am using a single node cluster for the moment)

Edit: I am testing this configuration by sending a bunch of ~1000 HTTP requests at the same time to the server.

Thank you in advance

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [March 1, 2021, 6:05pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057/2 "2021-03-01T18:05:51Z")

</div>

Hi @tim,

Thanks for reporting this. Could you please share more details about the error you’re running into? I think what you’re describing should work. One guess is maybe you don’t have 24 CPUs available on your machine so there aren’t enough resources for 4 replicas.

---

<div class="post-metadata">

**Author:** ![tim](https://avatars.discourse-cdn.com/v4/letter/t/ce73a5/32.png) [@tim](https://discuss.ray.io/u/tim)\
**Post date:** [March 2, 2021, 1:05pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057/3 "2021-03-02T13:05:55Z")

</div>

Hi,

Thank you a lot for your help it was just that: I assigned too much CPUs to each worker. Sorry about that.

Have a nice day,

Regards

---

<div class="post-metadata">

**Author:** ![tim](https://avatars.discourse-cdn.com/v4/letter/t/ce73a5/32.png) [@tim](https://discuss.ray.io/u/tim)\
**Post date:** [March 2, 2021, 1:49pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057/4 "2021-03-02T13:49:45Z")

</div>

Actually, I have another issue: I am not sure how to allocate the cpus/threads to my workers.

First of all, when I do htop I can see 15 cpus, but when I use only one replica and set num\_cpus to 15 it does not work, the only configuration that works is 13 cpus I don’t know why. It was suggested to use ray.nodes() to find insights but doing that I can clearly see that ray finds 15 cpus.

Second of all, I don’t know how to set OMP\_NUM\_THREADS in the following command: OMP\_NUM\_THREADS=16 ray start --head

It is said [here](https://docs.ray.io/en/master/configure.html) that “to avoid performance degradation with many workers” it should be set to 1, but I have one worker and 1 to 3 replicas, should I increase the number of threads then ?

Thank you for your help

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [March 2, 2021, 4:59pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057/5 "2021-03-02T16:59:58Z")

</div>

Hi Tim, sorry about that, the reason you can only fit 13 replicas is that two CPUs are already being used internally by Ray Serve, one for the Serve Controller actor and one for the HTTP proxy actor–we should probably make this more clear somewhere. In general, you can see what’s using your CPUs by looking at the Ray Dashboard.

I’m not sure the best answer to the OMP\_NUM\_THREADS question, @simon-mo do you know?

---

<div class="post-metadata">

**Author:** ![tim](https://avatars.discourse-cdn.com/v4/letter/t/ce73a5/32.png) [@tim](https://discuss.ray.io/u/tim)\
**Post date:** [March 4, 2021, 4:48pm UTC](https://discuss.ray.io/t/not-sure-how-num-replicas-works/1057/6 "2021-03-04T16:48:54Z")

</div>

thank you for your answer, it is very clear 🙂