# VLLM will report gpu missing on the hosting node in Ray

**URL:** <https://discuss.ray.io/t/vllm-will-report-gpu-missing-on-the-hosting-node-in-ray/21657>\
**Category:** Ray Clusters\
**Created:** [January 31, 2025, 3:53pm UTC](https://discuss.ray.io/t/vllm-will-report-gpu-missing-on-the-hosting-node-in-ray/21657 "2025-01-31T15:53:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pcpLiu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/pcpliu/32/6142_2.png) [@pcpLiu](https://discuss.ray.io/u/pcpLiu)\
**Post date:** [January 31, 2025, 3:53pm UTC](https://discuss.ray.io/t/vllm-will-report-gpu-missing-on-the-hosting-node-in-ray/21657/1 "2025-01-31T15:53:42Z")

</div>

# Context

We are using Ray Serve to deploy a vLLM App. It was working well til recently we upgrade version of vLLM to 0.7.0 and adapted to the API change.

We have a instance with six 4090 GPUs. We deployed a Ray cluster on it with one head node and 5 worker nodes. All are docker containers. Each container is attached to a gpu.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/8/8f6e9f130f7e3411131288284d2d74b41530afa0.png)

# Issue

The core issue is that whenever the vLLM app tries to load the model from the disk, it fails to find GPU to the container where the APP is hosted.

- All containers have exactly the same ENV
- We can use vllm cli to run the model directly without any issue.
- **We tried to re-deploy many times. Each time the APP could be hosted on an arbitrary node. Whenever a node becomes the hosting node, it will throw error of that it does not have GPU. But in the cases when that node is not the hosted node, it can work well and load model weights smoothly.**
- We are now trying to downgrade the vllm version but want to get an idea if this is a bug or our usage issue. Thanks!

# Log

```auto
 ValueError: Current node has no GPU available. current_node_resource={'node:172.17.0.6_group_0_5ea4bf00a38e2ed9e9af4d4e2c3d2c000000': 0.001, 'accelerator_type:G': 1.0, 'node:172.17.0.6_group_5ea4bf00a38e2ed9e9af4d4e2c3d2c000000': 0.001, 'CPU': 63.0, 'memory': 10593529856.0, 'object_store_memory': 4540084224.0, 'node:172.17.0.6': 0.999, 'bundle_group_0_5ea4bf00a38e2ed9e9af4d4e2c3d2c000000': 999.999, 'bundle_group_5ea4bf00a38e2ed9e9af4d4e2c3d2c000000': 999.999}. vLLM engine cannot start without GPU. Make sure you have at least 1 GPU available in a node current_node_id='6f6229bb91687736efbb6174c5885ad0ad7f5aa6fb53ad20afaee93a' current_ip='172.17.0.6'.

```

Full error log: [gist:0aea4772b3273a2e9a6427c77eb25354 · GitHub](https://gist.github.com/pcpLiu/0aea4772b3273a2e9a6427c77eb25354)

# Reproduce

Python package version

```auto
vllm==0.7.0
ray==2.41.0
ray[serve]==2.41.0

```

> <https://gist.github.com/pcpLiu/65fc7fd487c2afd91ed34b27d8eb5b0a>

---

<div class="post-metadata">

**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\
**Post date:** [February 3, 2025, 7:26pm UTC](https://discuss.ray.io/t/vllm-will-report-gpu-missing-on-the-hosting-node-in-ray/21657/2 "2025-02-03T19:26:32Z")

</div>

Hi! I’ve investigated this and talked to some of the Ray engineers and this is a known issue that they are currently working on fixing. In the meantime, can you try doing `distributed_executor_backend="mp"` and see if that fixes the issue?

---

<div class="post-metadata">

**Author:** ![pcpLiu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/pcpliu/32/6142_2.png) [@pcpLiu](https://discuss.ray.io/u/pcpLiu)\
**Post date:** [February 4, 2025, 1:03am UTC](https://discuss.ray.io/t/vllm-will-report-gpu-missing-on-the-hosting-node-in-ray/21657/3 "2025-02-04T01:03:40Z")

</div>

Thanks! I will try and report back the result