# Ray Serve LLM APIs **URL:** https://discuss.ray.io/c/ray-serve/llm-online-serving/29.md [Latest](https://discuss.ray.io/latest.md) · [Categories](https://discuss.ray.io/categories.md) · [Tags](https://discuss.ray.io/tags.md) --- ## [About the Ray Serve LLM APIs category](https://discuss.ray.io/t/about-the-ray-serve-llm-apis-category/22205)
**Author:** [@christina](https://discuss.ray.io/u/christina)\ **Replies:** 0\ **Last updated:** [April 2, 2025, 6:24pm UTC](https://discuss.ray.io/t/about-the-ray-serve-llm-apis-category/22205 "2025-04-02T18:24:58Z")
Ray Serve has LLM APIs to provide an easy way to deploy and scale multiple LLM models with a unified API. It supports automatic scaling, multi-model deployment, OpenAI-compatible endpoints, and LoRA multiplexing. The eng… --- ## [Setup api key to call LLM via rayserve](https://discuss.ray.io/t/setup-api-key-to-call-llm-via-rayserve/23319)
**Author:** [@AlaEddine](https://discuss.ray.io/u/AlaEddine)\ **Replies:** 15\ **Last updated:** [June 2, 2026, 11:41am UTC](https://discuss.ray.io/t/setup-api-key-to-call-llm-via-rayserve/23319 "2026-06-02T11:41:58Z")
Dear Ray community, I deployed kubeRay on kubernetes , and im serve my local LLMs , i want to add API Key when requesting my LLM, via rayserve service how i can setup api key ,is there an environment variable must be a… --- ## [How to route traffic to LiteLLM models using Serving LLMs](https://discuss.ray.io/t/how-to-route-traffic-to-litellm-models-using-serving-llms/22493)
**Author:** [@James\_Wong](https://discuss.ray.io/u/James_Wong)\ **Replies:** 8\ **Last updated:** [May 3, 2026, 10:29pm UTC](https://discuss.ray.io/t/how-to-route-traffic-to-litellm-models-using-serving-llms/22493 "2026-05-03T22:29:43Z")
We’re currently serving vLLM models via the ‘Serving LLMs’, Serving LLMs — Ray 2.46.0 Is there a way or an example to use Ray Serve to pass through LiteLLM models? We’re looking at the ‘Deploy Compositions of Models’ a… --- ## [Preprocessing in ray serve LLM](https://discuss.ray.io/t/preprocessing-in-ray-serve-llm/23346)
**Author:** [@manuel.ramblr](https://discuss.ray.io/u/manuel.ramblr)\ **Replies:** 3\ **Last updated:** [December 1, 2025, 11:50pm UTC](https://discuss.ray.io/t/preprocessing-in-ray-serve-llm/23346 "2025-12-01T23:50:06Z")
Hi everyone, I want to use ray serve LLM in the following way to host a model that is found on huggingface. In particular, I’m looking into this model: As you can see in their huggingface code, the preprocessing one… --- ## [Ray Serve vLLM multiple models per GPU in tensor parallelism](https://discuss.ray.io/t/ray-serve-vllm-multiple-models-per-gpu-in-tensor-parallelism/22973)
**Author:** [@LemurPwned](https://discuss.ray.io/u/LemurPwned)\ **Replies:** 1\ **Last updated:** [August 14, 2025, 6:37pm UTC](https://discuss.ray.io/t/ray-serve-vllm-multiple-models-per-gpu-in-tensor-parallelism/22973 "2025-08-14T18:37:52Z")
Hi! I was wondering if there’s a way to have multiple models request tensor parallelism across multi-gpu instances. For example, I have a simple 4xRTX5090 setup and I’d like to spread those two models over 4 GPUs, and … --- ## [vLLM v1 engine initialization workaround with vllm installation at runtime](https://discuss.ray.io/t/vllm-v1-engine-initialization-workaround-with-vllm-installation-at-runtime/22827)
**Author:** [@404notfound101](https://discuss.ray.io/u/404notfound101)\ **Replies:** 4\ **Last updated:** [July 20, 2025, 6:22am UTC](https://discuss.ray.io/t/vllm-v1-engine-initialization-workaround-with-vllm-installation-at-runtime/22827 "2025-07-20T06:22:52Z")
Hi team, I am working on using ray.serve.llm:build\_openai\_app to serve LLM on an EKS cluster. Due to permission issues, I am not allowed to modify the worker image to pre-install vllm, but have to include it through run… --- ## [How to log to stdout from Ray Serve](https://discuss.ray.io/t/how-to-log-to-stdout-from-ray-serve/22706)
**Author:** [@Shravani](https://discuss.ray.io/u/Shravani)\ **Replies:** 1\ **Last updated:** [June 23, 2025, 10:52am UTC](https://discuss.ray.io/t/how-to-log-to-stdout-from-ray-serve/22706 "2025-06-23T10:52:25Z")
We are using KubeRay in production as our inferencing solution. The logs are available for the deployment inside the metrics server of the head pod and ofcourse inside the filesystem of the pod at /tmp/ray/session\_\*/logs… --- ## [torch.distributed.DistNetworkError: The client socket has timed out after 600000ms while trying to connect to](https://discuss.ray.io/t/torch-distributed-distnetworkerror-the-client-socket-has-timed-out-after-600000ms-while-trying-to-connect-to/22557)
**Author:** [@arshadshaikh](https://discuss.ray.io/u/arshadshaikh)\ **Replies:** 3\ **Last updated:** [June 3, 2025, 4:59pm UTC](https://discuss.ray.io/t/torch-distributed-distnetworkerror-the-client-socket-has-timed-out-after-600000ms-while-trying-to-connect-to/22557 "2025-06-03T16:59:00Z")
1. Severity of the issue: (select one) High: Completely blocks me. 2. Environment: Ray version: 2.46.0 Python version: 3.10.16 OS: Ubuntu 22.04 LTS Cloud/Infrastructure: Azure VMSS Other libs/tools (if relevant): VL… --- ## [Ray Serve LLM APIs has 2~3x higher latency](https://discuss.ray.io/t/ray-serve-llm-apis-has-2-3x-higher-latency/22356)
**Author:** [@Jacob\_Chang](https://discuss.ray.io/u/Jacob_Chang)\ **Replies:** 7\ **Last updated:** [May 19, 2025, 8:50pm UTC](https://discuss.ray.io/t/ray-serve-llm-apis-has-2-3x-higher-latency/22356 "2025-05-19T20:50:23Z")
1. Severity of the issue: (select one) None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. \[V\] High: Com… --- ## [Ray Serve LLM example in document cannot work](https://discuss.ray.io/t/ray-serve-llm-example-in-document-cannot-work/22199)
**Author:** [@Hi\_u\_Bui\_Nguy\_n\_Trun](https://discuss.ray.io/u/Hi_u_Bui_Nguy_n_Trun)\ **Replies:** 6\ **Last updated:** [April 3, 2025, 7:21am UTC](https://discuss.ray.io/t/ray-serve-llm-example-in-document-cannot-work/22199 "2025-04-03T07:21:02Z")
Hi everyone, I’m a beginner of RayServe. I’m following the example of ray serve llm (https://docs.ray.io/en/latest/serve/llm/serving-llms.html) And it fails many ways. Here is my code (almost like the doc): from ray i…