ServeHandle with named deployment

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi,
I’m using deployments for models from a model registry (MLflow). I’m using the example code from the docs. I have multiple models, and my deployments are named.

@serve.deployment
class MLflowDeployment:
    def __init__(self, model_uri):
        self.model = mlflow.pyfunc.load_model(model_uri=model_uri)

    async def __call__(self, request):
        ...
        return self.model.predict(...)

MLflowDeployment.options(name="my_mlflow_model_1").deploy("model:/model1/1")
MLflowDeployment.options(name="my_mlflow_model_2").deploy("model:/model2/1")

I’d like to use the ServeHandle feature with those deployments.
How can I get the handle with named deployments? I looks like the get_handle method doesn’t have a name parameter. Did I miss something?

handle = MLflowDeployment.get_handle()  # missing name param?

By looking into the get_handle method, I see it uses the internal API. As a workaround, should I call it directly with my name?

get_global_client().get_handle(self._name, missing_ok=True, sync=sync)
get_global_client().get_handle("my_mlflow_model_1", missing_ok=True, sync=sync)

Thank you, Jonathan

EDIT: the same question applies to delete a deployment by name.

2

Try serve.get_deployment(NAME).get_handle()

https://docs.ray.io/en/latest/serve/core-apis.html#creating-a-deployment:~:text=You%20can%20also%20list%20all%20available%20deployments%20and%20dynamically%20get%20references%20to%20them%3A

3

Thanks, that’s what I was missing. It works well!