How do I best serve models with different docker images?

Ray Clusters Kubernetes
1

Hi, I’m a new user to ray serve and kuberay.

I have a use case where I would like to serve multiple models that are not necessarily related. E.g. A translation service, and a document embedding service, each with a different docker image. Is is possible to do so on one cluster?

I was thinking that I can either

  1. Use multiple applications e.g. Deploying Multiple Serve Applications — Ray 2.5.1
    But if I do this, then I also need each application to connect to workers that have the image loaded that contains the model. But I don’t know how to specify a different image per application.

  2. Have multiple RayServices handled by the same operator. I have tried to deploy multiple services with different names using the fruit example https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray_v1alpha1_rayservice.yaml , but any subsequent service after the first is in an unhealthy state according to the logs.

Also, if I have missed something obvious please let me know, and I apologize!

2

@Sihan_Wang @architkulkarni Any pointers what we can do with Ray Serve and KubeRay?