Continuous Delivery with Ray Serve

Ray Serve
1

Hi everyone. I have setup a ray cluster on a remote microk8s cluster, which also hosts a MLFlow model registry to store and track my models. To deploy my model to the cluster, I use ray serve to fetch my model from MLFlow (as here) and then and submit it to the cluster via ray submit
Now I would like to apply some basic continuous delivery. Ideally, a new model would be fetched from MLFlow and submitted to the cluster upon a commit to a master branch on GitHub.
Has anybody done something similar or could suggest some tools to facilitate this workflow?
Thanks!

2

Hi @Abaschkim, great question! The “best practice” here would probably be to have CI directly connect to the cluster via Ray Client Ray Client — Ray v2.0.0.dev0 and deploy the new endpoint. This is possible in Ray 1.3 which just got released! Other good options include running ray submit from CI, or making a Ray Serve endpoint that would trigger the operation of fetching/submitting, and then setting up a git hook to fire it off.

3

Thanks for the reply @architkulkarni. Great news! I will look into that. I am currently using a Kubernetes job as mentioned in the docs with Jenkins to deploy a ray serve instance.

4

Hi @Abaschkim mlflow deployment plugin for ray serve is a good starting point.

While the official plugin version doesn’t support remote deployment, there is an open PR that includes this behaviour.

1 Like