Multiple Serve instances on a ray cluster with serve REST API

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hey guys. I’m trying to serve multiple apps with REST API but they keep replacing each other. How can I deploy multiple different serve apps with REST API or any other programmatically available method?

2

Hi @Ali_Sayyah , for multiple apps, do you mean multiple FastAPI apps inside the same ray serve cluster?

Currently there is no support for mutiple fast apps, but you can use mount(Sub Applications - Mounts - FastAPI) to support under the same ray serve cluster.

3

Hi @Sihan_Wang . Thank you for your answer. No, I don’t use FastAPI. Just a simple starlette server. Is it still impossible?
I’m trying to deploy multiple separated apps with different prefixes to the serve cluster.

4

It should be working by using .deploy() API, not working in serve.run(). Let me know if you can’t work it out, and feel free to paste reproduce script. I can take a look!

Thank you!