# RuntimeError: CUDA error: invalid device ordinal when have multiple ray deployment on the same GPU

**URL:** <https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433>\
**Category:** Ray Serve\
**Created:** [April 28, 2023, 8:09pm UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433 "2023-04-28T20:09:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andcircle](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andcircle/32/4018_2.png) [@Andcircle](https://discuss.ray.io/u/Andcircle)\
**Post date:** [April 28, 2023, 8:09pm UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433/1 "2023-04-28T20:09:04Z")

</div>

I have a cluster with 8 GPU, trying to deploy multiple different models on it, each model will take a fraction of GPU

When start the service using any of following method:

1. serve.run(my\_class.options().bind(), name=name, route\_prefix=route)
2. driver = DAGDriver.bind(dag); serve.run(driver, port=8000)
3. serve run -p 8000 my\_file:my\_bind

Get error: RuntimeError: CUDA error: invalid device ordinal

Demo code which can reproduce the error:  
‘’'import torch  
import numpy as np  
import logging

import ray  
from ray import serve  
from starlette.requests import Request

import os

from ray.serve.drivers import DAGDriver

# Logging

logging.basicConfig(level=logging.DEBUG)

import torch  
import torch.nn as nn

class MyModel(nn.Module):  
def **init** (self, input\_size, hidden\_size, num\_classes):  
super(MyModel, self). **init** ()  
self.fc1 = nn.Linear(input\_size, hidden\_size)  
self.relu1 = nn.ReLU()  
self.fc2 = nn.Linear(hidden\_size, num\_classes)

```
def forward(self, x):
    out = self.fc1(x)
    out = self.relu1(out)
    out = self.fc2(out)
    return out

```

RAY\_ACTOR\_OPTIONS = {“num\_gpus”: 1}  
@serve.deployment(ray\_actor\_options=RAY\_ACTOR\_OPTIONS)  
class APP1:  
def **init** (self):

```
    print("@@@@@@@@@@@@@@@@ APP1 @@@@@@@@@@@@@@@@@@@@@@")
    print("ray.get_gpu_ids(): {}".format(ray.get_gpu_ids()))
    print("CUDA_VISIBLE_DEVICES: {}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
    
    device = f"cuda:{ray.get_gpu_ids()[0]}"
    print(f"Initializing APP1 to {device}")
    print("@@@@@@@@@@@@@@@@@ APP1 @@@@@@@@@@@@@@@@@@@@@")
    # device = torch.device(f"cuda:{ray.get_gpu_ids()[0]}")
    self.device = device
    self.model = MyModel(10, 10, 3)
    self.model.to(device)

async def __call__ (self, starlette_request: Request):
    input = (await starlette_request.json())["input"]
    input = np.array(input)

    output = self.model(torch.from_numpy(input).to(self.device))
    output = output.cpu().detach().numpy().tolist()
    
    return {"status":"success", "image":output}

```

RAY\_ACTOR\_OPTIONS = {“num\_gpus”: 1}  
@serve.deployment(ray\_actor\_options=RAY\_ACTOR\_OPTIONS)  
class APP2:  
def **init** (self):

```
    print("@@@@@@@@@@@@@@@@ APP2 @@@@@@@@@@@@@@@@@@@@@@")
    print("ray.get_gpu_ids(): {}".format(ray.get_gpu_ids()))
    print("CUDA_VISIBLE_DEVICES: {}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
    
    device = f"cuda:{ray.get_gpu_ids()[0]}"
    print(f"Initializing APP2 to {device}")
    print("@@@@@@@@@@@@@@@@@ APP2 @@@@@@@@@@@@@@@@@@@@@")
    # device = torch.device(f"cuda:{ray.get_gpu_ids()[0]}")
    self.device = device
    self.model = MyModel(10, 10, 3)
    self.model.to(device)

async def __call__ (self, starlette_request: Request):
    input = (await starlette_request.json())["input"]
    input = np.array(input)

    output = self.model(torch.from_numpy(input).to(self.device))
    output = output.cpu().detach().numpy().tolist()
    
    return {"status":"success", "image":output}

```

ray.init(num\_gpus=8)

# serve.run(APP1.options().bind(), name=“app1”, route\_prefix=f"/app1")

# serve.run(APP2.options().bind(), name=“app2”, route\_prefix=f"/app2")

dag = {  
‘/app1’: APP1.bind(),  
# ‘/image\_editor’: ImageEditor.bind(),  
# ‘/text2image’: Text2Image.bind(),  
‘/app2’: APP2.bind()  
}  
driver = DAGDriver.bind(dag)  
handle = serve.run(driver, port=8000)‘’’

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [May 3, 2023, 7:14am UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433/2 "2023-05-03T07:14:36Z")

</div>

Moving this to ray serve category.

cc: @Sihan_Wang

---

<div class="post-metadata">

**Author:** ![Andcircle](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andcircle/32/4018_2.png) [@Andcircle](https://discuss.ray.io/u/Andcircle)\
**Post date:** [May 3, 2023, 4:33pm UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433/3 "2023-05-03T16:33:56Z")

</div>

For some reason I can’t edit or delete this one any more, so I created a new one [when serve multiple models on a multi-gpu cluster, got error RuntimeError: CUDA error: invalid device ordinal](https://discuss.ray.io/t/when-serve-multiple-models-on-a-multi-gpu-cluster-got-error-runtimeerror-cuda-error-invalid-device-ordinal/10475)

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [May 3, 2023, 4:47pm UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433/4 "2023-05-03T16:47:01Z")

</div>

actually, I already moved the post to the right category for you. Let’s probably just keep one. Maybe let me delete this one?

---

<div class="post-metadata">

**Author:** ![Andcircle](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andcircle/32/4018_2.png) [@Andcircle](https://discuss.ray.io/u/Andcircle)\
**Post date:** [May 3, 2023, 8:07pm UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433/5 "2023-05-03T20:07:18Z")

</div>

Sure please delete this one, thanks

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [May 3, 2023, 8:19pm UTC](https://discuss.ray.io/t/runtimeerror-cuda-error-invalid-device-ordinal-when-have-multiple-ray-deployment-on-the-same-gpu/10433/6 "2023-05-03T20:19:10Z")

</div>