# Ray Serve container runtime\_env cannot use GPU

**URL:** <https://discuss.ray.io/t/ray-serve-container-runtime-env-cannot-use-gpu/10398>\
**Category:** Ray Serve\
**Created:** [April 27, 2023, 11:11am UTC](https://discuss.ray.io/t/ray-serve-container-runtime-env-cannot-use-gpu/10398 "2023-04-27T11:11:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![yhsmiley](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yhsmiley/32/4313_2.png) [@yhsmiley](https://discuss.ray.io/u/yhsmiley)\
**Post date:** [April 27, 2023, 11:11am UTC](https://discuss.ray.io/t/ray-serve-container-runtime-env-cannot-use-gpu/10398/1 "2023-04-27T11:11:34Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

I am trying to run a ray serve deployment with docker containers, but I am not able to get the replicas to use gpu. I am running on a local setup with 1 gpu.

docker-compose file for ray head:

```auto
    ray_head:
        image: ray_head
        container_name: ray_head
        build:
            context: ./ray_pipeline/ray_head
            dockerfile: Dockerfile
        command: ray start --head --dashboard-host=0.0.0.0 --block
        ports:
            - 8265:8265 # ray dashboard
            - 8000:8000 # ray serve
            - 10001:10001 # ray client
        restart: always
        privileged: true
        volumes:
            - /dev/shm:/dev/shm
            - /var/lib/containers:/var/lib/containers
        environment:
            - NVIDIA_VISIBLE_DEVICES=all
        networks:
            - ray_network

```

Snippet for ray serve deployment:

```auto
import ray

runtime_env = {
    "container": {
        "image": "detector:latest",
        "run_options": ["--gpus all", "-v /dev/shm:/dev/shm", "--privileged", "--log-level=debug"]
    }
}
@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 1, "num_gpus": 0.25, "runtime_env": runtime_env})
class RayDetector(object):
    def __init__ (self):
        import os
        import torch
        print(f'# ray.get_gpu_ids(): {ray.get_gpu_ids()}')
        print(f'# os.environ["CUDA_VISIBLE_DEVICES"]: {os.environ["CUDA_VISIBLE_DEVICES"]}')
        print(f'# torch.cuda.is_available(): {torch.cuda.is_available()}')
        
        # initialize object detector

```

Snippet from main code:

```auto
import ray
from ray import serve

ray.init(address='ray://localhost:10001')
serve.start(detached=True, http_options={"host": "0.0.0.0"})

RayDetector.deploy()
ray_detector_handle = serve.get_deployment('RayDetector').get_handle()

```

When I run the above code, I get the below:

```auto
ray.get_gpu_ids(): [0]
os.environ["CUDA_VISIBLE_DEVICES"]: 0
torch.cuda.is_available(): False

```

What could be the issue here? It seems that the gpu device is properly detected, but pytorch is not able to use it.

---

<div class="post-metadata">

**Author:** ![shrekris](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shrekris](https://discuss.ray.io/u/shrekris)\
**Post date:** [June 5, 2023, 6:25pm UTC](https://discuss.ray.io/t/ray-serve-container-runtime-env-cannot-use-gpu/10398/2 "2023-06-05T18:25:56Z")

</div>

Is there any chance this is an issue with the Docker container itself? What do you see if you run the container and manually check `torch.cuda.is_available()` in the Python interpreter?

---

<div class="post-metadata">

**Author:** ![yhsmiley](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yhsmiley/32/4313_2.png) [@yhsmiley](https://discuss.ray.io/u/yhsmiley)\
**Post date:** [June 14, 2023, 5:43am UTC](https://discuss.ray.io/t/ray-serve-container-runtime-env-cannot-use-gpu/10398/3 "2023-06-14T05:43:25Z")

</div>

I have solved this by installing nvidia-docker2 inside the ray head container.

Now I am running into another problem. On the dashboard ‘cluster’ page, I am not able to see the status (e.g. pending tasks, object refs in scope) of the replicas that are spinned up using podman.

---

<div class="post-metadata">

**Author:** ![psydok](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/psydok/32/3518_2.png) [@psydok](https://discuss.ray.io/u/psydok)\
**Post date:** [December 6, 2023, 6:02am UTC](https://discuss.ray.io/t/ray-serve-container-runtime-env-cannot-use-gpu/10398/4 "2023-12-06T06:02:46Z")

</div>

Could you please tell me, did you manage to pull up the service image and run the service in a container this way?