# Does RayData Support multi-node vllm inference **URL:** \ **Category:** Ray Data LLM APIs\ **Created:** [May 20, 2025, 12:10pm UTC](https://discuss.ray.io/t/does-raydata-support-multi-node-vllm-inference/22530 "2025-05-20T12:10:07Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![DriverSong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/driversong/32/7980_2.png) [@DriverSong](https://discuss.ray.io/u/DriverSong)\ **Post date:** [May 20, 2025, 12:10pm UTC](https://discuss.ray.io/t/does-raydata-support-multi-node-vllm-inference/22530/1 "2025-05-20T12:10:08Z")
**1. Severity of the issue: (select one)** None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. High: Completely blocks me. **2. Environment:** - Ray version: 2.43.0 - Python version: 3.10.12 - OS: Ubuntu 22.04.4 LTS - Cloud/Infrastructure: 8\*L40 per node of k8s - Other libs/tools (if relevant): - vllm version: 0.8.5 **3. What happened vs. what you expected:** - Expected: I attempt to use RayData to create `deepseek-r1` inference job by `vllm` with 16 \* L40 with 8 \* L40 per node. - Actual: While I set the `num_gpus` of `map_batches` to `16`, the cluster return the res: ```auto (autoscaler +44s) Error: No available node types can fulfill resource request {'GPU': 16.0, 'CPU': 1.0}. Add suitable node types to this cluster to resolve this issue. (autoscaler +1m19s) Error: No available node types can fulfill resource request {'CPU': 1.0, 'GPU': 16.0}. Add suitable node types to this cluster to resolve this issue. ``` While the max gpu count per actor is limited by the node size, there will never be an actor with 16 gpu. Then, I change the `num_gpus` of `map_batches` to `0`, execute the vllm main process in the map\_batches actor with only cpu, and got the following Error: ```auto (MapWorker(MapBatches(SubClass)) pid=41596, ip= **** ) INFO 05-20 18:19:06 [config.py:717] This model supports multiple tasks: {'embed', 'score', 'generate', 'classify', 'reward'}. Defaulting to 'generate'. (MapWorker(MapBatches(SubClass)) pid=41596, ip= **** ) INFO 05-20 18:19:07 [arg_utils.py:1669] Engine in background thread is experimental on VLLM_USE_V1=1. Falling back to V0 Engine. (MapWorker(MapBatches(SubClass)) pid=41596, ip= **** ) INFO 05-20 18:19:07 [config.py:1770] Defaulting to use ray for distributed inference (MapWorker(MapBatches(SubClass)) pid=41596, ip= **** ) WARN 05-20 18:19:07 [fp8.py:63] Detected fp8 checkpoint. Please note that the format is experimental and subject to change. (MapWorker(MapBatches(SubClass)) pid=41596, ip= **** ) Failed to initialize model or tokenizer: 'NoneType' object is not subscriptable ``` Ray’s resource isolation makes GPU resources invisible to the main process of vllm, causing vllm initialization failure. Finally, I change the `num_gpus` of `map_batches` to `1` and get the following log ```auto (MapWorker(MapBatches(SubClass)) pid=42405, ip=****) INFO 05-20 18:34:09 [ray_utils.py:233] Waiting for creating a placement group of specs for 10 seconds. specs=[{'GPU': 1.0, 'node:****': 0.001}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` and `ray list nodes` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node. ``` This time, vllm is pending when applying for placemant\_group. This is because the entire cluster has only 16 \* L40. While the vllm main process occupies one card, only 15 \* L40 available. Does RayData not support the multi-mode inference of vllm? or one card must be wasted for the initialization of vllm with RayData. btw, I also try RayServe to solve this problem, while it also has the arg `num_gpus`, and I guess it will not work. Will Ray support this scenarios? Or anyone have similar demand scenarios with solutions? At present, I can only think of building an additional platform on top of ray (kuberay or k8s-operator) to achieve the management of multi-node vllm inference tasks on ray. ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [May 22, 2025, 6:59pm UTC](https://discuss.ray.io/t/does-raydata-support-multi-node-vllm-inference/22530/2 "2025-05-22T18:59:45Z")
Did you try Ray Data LLM? [Working with LLMs — Ray 3.0.0.dev0](https://docs.ray.io/en/master/data/working-with-llms.html) ---
**Author:** ![kourosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kourosh/32/2130_2.png) [@kourosh](https://discuss.ray.io/u/kourosh)\ **Post date:** [May 23, 2025, 8:14pm UTC](https://discuss.ray.io/t/does-raydata-support-multi-node-vllm-inference/22530/3 "2025-05-23T20:14:14Z")
> [@DriverSong](#): > > Will Ray support this scenarios? Or anyone have similar demand scenarios with solutions? At present, I can only think of building an additional platform on top of ray (kuberay or k8s-operator) to achieve the management of multi-node vllm inference tasks on ray. On Ray serve side, this example should be a good starting point. [Serve DeepSeek — Ray 2.46.0](https://docs.ray.io/en/latest/serve/tutorials/serve-deepseek.html)