What is best practice for local setup?

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

We have a team that would like to use Ray serve to set up an ecosystem of models behind a web app. We are having difficulty getting a deployment for local development.

  1. When we try to use docker-compose with ray images we run into problems that the ray image python version (3.7) is not compatible with the web server’s version (3.9) which is needed for running fastAPI with all the pydantic sugar.

  2. For local development we would like to be able to launch a few containers with ray running, e.g. a head and some workers, then deploy our fastAPI webserver on top of ray serve. This is our docker compose file:

services:
  ray-head:
    image: ray_head
    build:
      context: ./
      dockerfile: ray_head/Dockerfile
    ports:
      - 8265:8265
      - 8000:8000
    volumes:
      - /dev/shm:/dev/shm
    networks:
      - safe-tier
  app:
    image: app
    build:
      context: ./
      dockerfile: app/Dockerfile
    depends_on:
      - ray-head
    networks:
      - safe-tier
networks:
  safe-tier:

where ray_head/Dockerfile entry point is:

ENTRYPOINT [ "ray", "start", "--head", "--port=6379", "--redis-shard-ports=6380,6381", "--object-manager-port=22345","--node-manager-port=22346","--dashboard-host=0.0.0.0","--block"]

where the app’s main entry point contains:

import ray
from ray import serve

from external_app import External

if __name__ == '__main__':

    ray.init(address='ray://ray-head:10001')
    serve.start(detached=True)

    External.deploy()

This seems to launch the app, as it’s visible on the ray dashboard but then it’s not available at port 8000 on localhost despite being exposed to the host.

app-1  | (ServeController pid=256) INFO 2022-06-14 22:28:59,993 controller 256 deployment_state.py:1216 - Adding 1 replicas to deployment 'External'.
app-1  | INFO:/usr/local/lib/python3.9/site-packages/ray/serve/client.py:Deployment 'External' is ready at `http://127.0.0.1:8000/api1`. component=serve deployment=External

How do we access the deployed app?

2

Can you change serve.start(detached=True) to serve.start(detached=True, http_options={"host": "0.0.0.0"}? This should expose your deployment publicly, so you can access it using http://$RAY_HEAD_SERVICE_HOST:8000/api1. There’s more info in this documentation.

3

That was the solution. Thank you! With this was able to dockerise the setup and gain access.

4

For anyone in the future interested we use this setup for local development.

A docker compose file:

services:
  # bunch of other containers
  ray-head:
    image: ray_head
    container_name: ray-head
    build:
      context: ./
      dockerfile: packagename/deployments/ray_head/Dockerfile
    ports:
      - 8265:8265
      - 8000:8000
    privileged: true
    volumes:
      - /dev/shm:/dev/shm
    networks:
      - safe-tier
networks:
  safe-tier:

A launch script:

sudo docker compose down
sudo docker compose build
sudo docker compose up ray-head -d
# launch other containers similarly
sleep 10
sudo docker exec -it ray-head bash -c 'python /code/deployments/app/main.py'

These lines when connecting to ray instance:

ray.init(address='ray://ray-head:10001')
serve.start(detached=True,  http_options={"host": "0.0.0.0"})

This is the ray-head Dockerfile:

FROM python:3.9

WORKDIR /code

# Do this separatly to use cache
COPY ./packagename/requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

# will change frequently
COPY ./packagename /code/packagename
RUN pip install /code/packagename

ENTRYPOINT [ "ray", "start", "--head", "--port=6379", "--redis-shard-ports=6380,6381", "--object-manager-port=22345","--node-manager-port=22346","--dashboard-host=0.0.0.0","--block"]
2 Likes
5

The code above saved me hours of figuring everything out myself, much appreciated!