Setup api key to call LLM via rayserve

Ray Serve Ray Serve LLM APIs
1

Dear Ray community,

I deployed kubeRay on kubernetes , and im serve my local LLMs , i want to add API Key when requesting my LLM, via rayserve service

how i can setup api key ,is there an environment variable must be added to the RayService config , what is the option for that,

call my llm via http://ray-serve-llm-serve-svc_ip:8000 , i expose this service via ingress nginx for external access

2

Ray Serve LLM does not natively enforce API key authentication via an environment variable or built-in config. The VLLM_API_KEY environment variable is ignored by Ray Serve LLM; authentication must be implemented at the application layer, such as by adding FastAPI middleware to your Ray Serve deployment. This is different from vanilla vLLM, which supports VLLM_API_KEY directly. For production, you should add authentication middleware to your FastAPI app before exposing it externally via ingress nginx, as Ray Serve does not provide this out of the box according to Ray GitHub Issue #55369.

Would you like more detail or a code example for adding API key authentication middleware to your Ray Serve FastAPI app?

Sources:

Hint: Mention @RunLLM in the post for followups.

3

thanks for the feedback

can you provide me more API key authentication middleware to your Ray Serve , how i can add to the existing deployment, must re-build the rayserve docker image, what is the best solution