I’m following the quickstart example given at Quickstart examples — Ray 2.53.0 for deploying Ray serve applications.
For one of my deployments, I have a unique requirement to download the model on the fly from a proprietary data store from each worker pod. I’m unable to replicate the model into S3 or shared file systems.
Does Ray Serve support injecting custom download logic into every worker before starting Vllm?
Yes, Ray Serve LLM supports injecting custom download logic into every worker before starting vLLM by using the callback API. You can define a custom callback class (subclassing CallbackBase) and specify it in the callback_config of your LLMConfig. This callback’s on_before_node_init method will be invoked on each worker node before model initialization, allowing you to implement proprietary download logic (e.g., fetching from your data store) as needed. See the design and usage details in this merged PR and the callback base class implementation here.
Would you like a step-by-step example of how to implement and configure such a callback?
Sources:
Hint: Mention @RunLLM in the post for followups.
@RunLLM Why is CallbackBase defined in _internal module if it’s meant to be used by end users?
Currently, CallbackBase is defined in the _internal module because the callback API is still considered experimental and not yet a stable public API. The Ray team is gathering user feedback and may move it to a public location with full documentation in the future, as discussed in this issue. For now, advanced users can use it, but the API may change.
Would you like more detail on the roadmap or how to safely use it in your project?
Sources:
Hint: Mention @RunLLM in the post for followups.
Hey Nilesh, RunLLM is correct that we have plans to make the Callbacks API stable and public in around a month. Feel free to try out the current internal module and let me know if you run into any blockers!
Thank you, Jeffrey!
I don’t have any blockers right now but I’m happy to help in any way I can.