# How to run multiple deployments in ray serve 2.0

**URL:** <https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681>\
**Category:** Ray Serve\
**Created:** [September 26, 2022, 5:27pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681 "2022-09-26T17:27:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![artemisart](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/artemisart/32/3019_2.png) [@artemisart](https://discuss.ray.io/u/artemisart)\
**Post date:** [September 26, 2022, 5:27pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/1 "2022-09-26T17:27:39Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

Hello, how is it possible to deploy multiple models which are not in the same graph? i.e. I want to deploy A and B and query them with url/a, url/b.  
I tried using multiple `serve.run()` or `serve run file:deployment`, but new runs always remove the previous deployment.  
Also the RayService config has only one `import_part`, how can we deploy multiple services in one RayService?  
Thanks

---

<div class="post-metadata">

**Author:** ![shrekris](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shrekris](https://discuss.ray.io/u/shrekris)\
**Post date:** [September 26, 2022, 5:47pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/2 "2022-09-26T17:47:16Z")

</div>

You can create independent deployments and bind them to a single `DAGDriver`. Then, you can `serve.run()`/`serve run` that single `DAGDriver`.

[This section](https://docs.ray.io/en/releases-2.0.0/serve/migration.html#migrate-deployments-with-route-prefixes) in the docs provides more details.

---

<div class="post-metadata">

**Author:** ![PaulRudin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/paulrudin/32/3235_2.png) [@PaulRudin](https://discuss.ray.io/u/PaulRudin)\
**Post date:** [September 27, 2022, 7:03am UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/3 "2022-09-27T07:03:29Z")

</div>

Yeah, although a single DAGDriver is a bit problematic if you have multiple endpoints (last example on the page linked above) and at least one of them has a fastapi ingress - the paths don’t work out as you’d expect.

---

<div class="post-metadata">

**Author:** ![PaulRudin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/paulrudin/32/3235_2.png) [@PaulRudin](https://discuss.ray.io/u/PaulRudin)\
**Post date:** [September 27, 2022, 7:09am UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/4 "2022-09-27T07:09:22Z")

</div>

Incidentally: the dictionary form for constructing a DAGDriver isn’t mentioned in the general purpose docs for 2.0 (as far as I can see) only in the migration guide.

---

<div class="post-metadata">

**Author:** ![puntime\_error](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/puntime_error/32/502_2.png) [@puntime\_error](https://discuss.ray.io/u/puntime_error)\
**Post date:** [October 4, 2022, 4:15pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/5 "2022-10-04T16:15:05Z")

</div>

When I was converting from ray 1.11 to 2.0, I was still able to deploy multiple models like this:

`Model1.options(init_args=model1_args, num_replicas=1, name="model1").deploy()`

`Model2.options( init_args=model2_args, num_replicas=num_reps, name="model2", ray_actor_options={"num_gpus": 1} ).deploy()`

That being said, I was making direct calls on the model and not via the url served endpoints.

I did eventually convert over to use the DAG just to see how to works

---

<div class="post-metadata">

**Author:** ![vakker00](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vakker00](https://discuss.ray.io/u/vakker00)\
**Post date:** [October 9, 2022, 12:55pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/6 "2022-10-09T12:55:23Z")

</div>

@PaulRudin I’m not sure about the “the paths don’t work out as you’d expect”, do you have an example?

For me these two options work:

**Option 1:**

```python
import requests
from fastapi import FastAPI
from ray import serve

app = FastAPI()

@serve.deployment()
@serve.ingress(app)
class Serve:
    @app.get("/ping1")
    def ping1(self):
        return "pong1"

    @app.get("/ping2")
    def ping2(self):
        return "pong2"

serve.run(Serve.bind())
resp = requests.get("http://localhost:8000/ping1")
assert resp.json() == "pong1"
resp = requests.get("http://localhost:8000/ping2")
assert resp.json() == "pong2"

```

**Option 2:**

```python
import requests
from ray import serve
from ray.serve.drivers import DAGDriver

@serve.deployment()
class Ping1:
    def __call__ (self):
        return "pong1"

@serve.deployment()
class Ping2:
    def __call__ (self):
        return "pong2"

d = DAGDriver.bind({"/ping1": Ping1.bind(), "/ping2": Ping2.bind()})
handle = serve.run(d)

resp = requests.get("http://localhost:8000/ping1")
assert resp.json() == "pong1"
resp = requests.get("http://localhost:8000/ping2")
assert resp.json() == "pong2"

```

**For reference, this doesn’t work:**

```python
import requests
from fastapi import FastAPI
from ray import serve

@serve.deployment(route_prefix="/ping")
class Ping1:
    def __call__ (self):
        return "pong1"

@serve.deployment(route_prefix="/ping")
class Ping2:
    def __call__ (self):
        return "pong2"

serve.run(Ping1.bind())
serve.run(Ping2.bind())

resp = requests.get("http://localhost:8000/ping1")
assert resp.json() == "pong1"
resp = requests.get("http://localhost:8000/ping2")
assert resp.json() == "pong2"

```

---

<div class="post-metadata">

**Author:** ![PaulRudin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/paulrudin/32/3235_2.png) [@PaulRudin](https://discuss.ray.io/u/PaulRudin)\
**Post date:** [October 11, 2022, 11:43am UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/7 "2022-10-11T11:43:48Z")

</div>

See here:

> [@Routing with DAGDriver and serve.ingress](https://discuss.ray.io/t/routing-with-dagdriver-and-serve-ingress/7665/3):
>
> Thanks. I understand that works, but I specifically am interested with the case with both DAGDriver and @serve.ingress, because I want to be able to use FastApi wildcard routes, but also have some deployments that can be scaled independently. The question is really what specifically goes wrong with my example - why don’t we just get a 200 with “hello” as the response body?

---

<div class="post-metadata">

**Author:** ![lizzzcai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lizzzcai/32/3417_2.png) [@lizzzcai](https://discuss.ray.io/u/lizzzcai)\
**Post date:** [November 14, 2022, 1:13am UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/8 "2022-11-14T01:13:16Z")

</div>

Hi Ray Serve team, I would like to check if ray serve currently supports to use case mentioned in the question? To deploy multiple models via multiple `serve run` on different URL\_Path, like: `serve run model1:deployment` on `/model1` and `serve run model2:deployment` on `/model2`. Thanks.

---

<div class="post-metadata">

**Author:** ![Sihan\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sihan_wang/32/2570_2.png) [@Sihan\_Wang](https://discuss.ray.io/u/Sihan_Wang)\
**Post date:** [November 14, 2022, 5:07pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/9 "2022-11-14T17:07:03Z")

</div>

Hi @lizzzcai ,

As the discuss mentioned, right now you can only be able to deploy multiple models with single `serve run` in 2.0 ([1.x to 2.x API Migration Guide — Ray 2.0.0](https://docs.ray.io/en/releases-2.0.0/serve/migration.html#migrate-deployments-with-route-prefixes)).  
The old api is still supported until we provide future api to use for your use case. ([How to run multiple deployments in ray serve 2.0 - #5 by puntime\_error](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/5))

---

<div class="post-metadata">

**Author:** ![lizzzcai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lizzzcai/32/3417_2.png) [@lizzzcai](https://discuss.ray.io/u/lizzzcai)\
**Post date:** [November 15, 2022, 9:23am UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/10 "2022-11-15T09:23:55Z")

</div>

Thanks @Sihan_Wang for your reply. Just want to know that is it something in the roadmap or it will need to open up an issue, thanks.

---

<div class="post-metadata">

**Author:** ![Sihan\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sihan_wang/32/2570_2.png) [@Sihan\_Wang](https://discuss.ray.io/u/Sihan_Wang)\
**Post date:** [December 13, 2022, 10:06pm UTC](https://discuss.ray.io/t/how-to-run-multiple-deployments-in-ray-serve-2-0/7681/11 "2022-12-13T22:06:18Z")

</div>

Hi, we just publish a [RFC](https://github.com/ray-project/ray/issues/31072) about the multi applications API.  
Welcome comments & suggestions! Thank you!