About the Ray Serve LLM APIs category

[christina]

Ray Serve has LLM APIs to provide an easy way to deploy and scale multiple LLM models with a unified API. It supports automatic scaling, multi-model deployment, OpenAI-compatible endpoints, and LoRA multiplexing. The engine-agnostic architecture works with frameworks like vLLM and SGLang, enabling efficient model serving across multiple nodes.

More documentation:

• https://docs.ray.io/en/latest/serve/llm/serving-llms.html