How to deploy Aviary Frontend along with Backend in the same Ray cluster?

LLMs/Generative AI/Aviary
1

Hi guys,

I got the backend running using deploy/ray/aviary-cluster.yaml. And I got Aviary Frontend running locally using AVIARY_URL. Now I am trying to “deploy” Aviary Frontend along with the backend in the same cluster.

I don’t know Ray Cluster / Ray Serve. I saw some documentation about multi-app deploy, but could not make sense of how the backend should be set in the config file.

I tried the following:

ray attach my-aviary-cluster.yaml

serve build --multi-app aviary.backend:llm_application aviary.frontend.app:app -o app_config.yaml

and got:

TypeError: Expected 'aviary.backend:llm_application' to be an Application but got <class 'function'>.

Thanks in advance for your help.

2

Hey @cirocavani, you can directly use the python API and ray serve to run both the backend and the frontend on the same ray cluster as shown below.

This snippet will:

  1. Initialize ray to set environment variables
  2. start the backend at localhost:8000/backend
  3. start the frontend at localhost:8000/frontend

To test it, you can run ray attach -p 8000 my-aviary-cluster.yaml, which should forward port 8000 to your laptop. You should be able to view the frontend at localhost:8000/frontend on your laptop.

Hope it helps!

# initialize ray
import ray
ray.init(runtime_env={"env_vars": {"AVIARY_URL": "http://localhost:8000/backend"}})

# Run the backend
from ray import serve
from aviary.backend.server.run import llm_server

models = ["models/amazon--LightGPT.yaml"]
serve.run(llm_server(models), name="backend", route_prefix="/backend")

# Run the frontend
from aviary.frontend.app import app
serve.run(app, name="frontend", route_prefix="/frontend")
1 Like