How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Hello, how is it possible to deploy multiple models which are not in the same graph? i.e. I want to deploy A and B and query them with url/a, url/b. I tried using multiple serve.run() or serv…

You can create independent deployments and bind them to a single DAGDriver. Then, you can serve.run()/serve run that single DAGDriver. This section in the docs provides more details.

Yeah, although a single DAGDriver is a bit problematic if you have multiple endpoints (last example on the page linked above) and at least one of them has a fastapi ingress - the paths don’t work out as you’d expect.

Incidentally: the dictionary form for constructing a DAGDriver isn’t mentioned in the general purpose docs for 2.0 (as far as I can see) only in the migration guide.

When I was converting from ray 1.11 to 2.0, I was still able to deploy multiple models like this: Model1.options(init_args=model1_args, num_replicas=1, name="model1").deploy() Model2.options( init_args=model2_args, num_replicas=num_reps, name="model2", ray_actor_options={"num_gpus": 1} …

@PaulRudin I’m not sure about the “the paths don’t work out as you’d expect”, do you have an example? For me these two options work: Option 1: import requests from fastapi import FastAPI from ray import serve app = FastAPI() @serve.deployment() @serve.ingress(app) class Serve: @app.get("/…

See here: [image] Routing with DAGDriver and serve.ingress Ray Serve Thanks. I understand that works, but I specifically am interested with the case with both DAGDriver and @serve.ingress, because I want to be able to use FastApi wildcard routes, but also have some deploym…

Hi Ray Serve team, I would like to check if ray serve currently supports to use case mentioned in the question? To deploy multiple models via multiple serve run on different URL_Path, like: serve run model1:deployment on /model1 and serve run model2:deployment on /model2. Thanks.

Hi @lizzzcai , As the discuss mentioned, right now you can only be able to deploy multiple models with single serve run in 2.0 ( 1.x to 2.x API Migration Guide — Ray 2.0.0 ). The old api is still supported until we provide future api to use for your use case. ( How to run multiple deployments in ray …

Thanks @Sihan_Wang for your reply. Just want to know that is it something in the roadmap or it will need to open up an issue, thanks.

Hi, we just publish a RFC about the multi applications API. Welcome comments & suggestions! Thank you!

How to run multiple deployments in ray serve 2.0

Ray Serve

PaulRudin October 11, 2022, 11:43am 7

See here:

Topic		Replies	Views
About Ray DAG API for serve.deployment at Ray 2.44.1 Ray Serve	4	98	December 9, 2025
Multiple Serve instances on a ray cluster with serve REST API Ray Serve	3	688	November 30, 2022
Deploy, delete and use deployments in Ray Serve 2.0.0 Ray Serve	8	2233	December 13, 2022
Automating the serving of many different models Ray Serve	8	2001	May 3, 2023
Production best practices for Ray Serve Ray Serve	6	1367	August 15, 2023

How to run multiple deployments in ray serve 2.0

Related topics