# Parallel Detectron2(Pytorch) inference with GPU

**URL:** <https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267>\
**Category:** Ray Core\
**Created:** [May 25, 2022, 6:15pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267 "2022-05-25T18:15:36Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![james811223](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/james811223/32/2604_2.png) [@james811223](https://discuss.ray.io/u/james811223)\
**Post date:** [May 25, 2022, 6:15pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/1 "2022-05-25T18:15:36Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

**Problem: I’m unable to parallelize a function.**

**What the function does:**

1. Some stuff
2. Load image from AWS S3.
3. Preprocess image
4. Make inference with a detectron2 model. (GPU)
5. Apply rules to the output of the model inference.
6. Return data

Does anyone know how to parallelize this function?

---

<div class="post-metadata">

**Author:** ![Mingwei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mingwei/32/997_2.png) [@Mingwei](https://discuss.ray.io/u/Mingwei)\
**Post date:** [May 25, 2022, 6:44pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/2 "2022-05-25T18:44:41Z")

</div>

Do you want to have parallelized execution of many instances of this function over many images, or parallelize certain steps within this function?

---

<div class="post-metadata">

**Author:** ![Mingwei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mingwei/32/997_2.png) [@Mingwei](https://discuss.ray.io/u/Mingwei)\
**Post date:** [May 25, 2022, 10:16pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/3 "2022-05-25T22:16:34Z")

</div>

Assume the function takes an S3 URL, and returns data. You probably can parallelize running multiple functions on a list of S3 URLs (`s3_url_list`):

```auto
@ray.remote
def process_image(s3_url):
    ......

results = [process_image.remote(url) for url in s3_url_list]

```

---

<div class="post-metadata">

**Author:** ![james811223](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/james811223/32/2604_2.png) [@james811223](https://discuss.ray.io/u/james811223)\
**Post date:** [May 26, 2022, 12:57pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/4 "2022-05-26T12:57:10Z")

</div>

I need to parallelized the whole function. I tried many different ways. The processes just don’t recognize the GPUs.

---

<div class="post-metadata">

**Author:** ![james811223](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/james811223/32/2604_2.png) [@james811223](https://discuss.ray.io/u/james811223)\
**Post date:** [May 26, 2022, 1:10pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/5 "2022-05-26T13:10:01Z")

</div>

Here’s my code:

```auto
Imports .....

def fun_a_for_fun_to_parallel(...):
    ...

def fun_b_for_fun_to_parallel(...):
    ...

def fun_c_for_fun_to_parallel(...):
    ...

@ray.remote
def fun_to_parallel(...):
    stuff...
    function call with model inference...
    stuf...
    return ...

```

One of the other scripts being imported into main script above:

```auto
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor
more imports

cfg = get_cfg()
more configs
model = DefaultPredictor(cfg)

def fun(...):
    stuff
    outputs = model(im)
    stuff
    return ...

```

I’ve also tried replicating the model and assigned them to different GPUs(cuda:0, 1, 2…), which didn’t work.

I tried setting ngpus to like 2, 3, 4… for ray init.

---

<div class="post-metadata">

**Author:** ![Mingwei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mingwei/32/997_2.png) [@Mingwei](https://discuss.ray.io/u/Mingwei)\
**Post date:** [May 26, 2022, 6:31pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/6 "2022-05-26T18:31:18Z")

</div>

Have you tried setting `num_gpus=1` when converting a function to Ray remote function? e.g.

```auto
@ray.remote(num_gpus=1)
def fun_to_parallel(...):
    ...

```

More documentations are at [GPU Support — Ray 1.12.1](https://docs.ray.io/en/latest/ray-core/tasks/using-ray-with-gpus.html)

---

<div class="post-metadata">

**Author:** ![Mingwei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mingwei/32/997_2.png) [@Mingwei](https://discuss.ray.io/u/Mingwei)\
**Post date:** [May 26, 2022, 9:46pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/7 "2022-05-26T21:46:50Z")

</div>

Another way to convert a function to Ray remote function, without `@ray.remote` decorator, is by calling `ray.remote(func).options(num_cpus=xx).remote(args...)`

---

<div class="post-metadata">

**Author:** ![james811223](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/james811223/32/2604_2.png) [@james811223](https://discuss.ray.io/u/james811223)\
**Post date:** [June 8, 2022, 3:00pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/8 "2022-06-08T15:00:52Z")

</div>

I tried using Ray Serve, and it’s now working. Thanks @Mingwei for your help!

---

<div class="post-metadata">

**Author:** ![Sujit\_Kumar](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sujit_kumar/32/3018_2.png) [@Sujit\_Kumar](https://discuss.ray.io/u/Sujit_Kumar)\
**Post date:** [August 16, 2022, 1:11pm UTC](https://discuss.ray.io/t/parallel-detectron2-pytorch-inference-with-gpu/6267/9 "2022-08-16T13:11:44Z")

</div>

@Mingwei @james811223  
I am unable to use multi gpu while doing inference. I have raised an issue at [Issue on page /serve/getting\_started.html · Issue #27905 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/27905)  
can you please help?