We have a team that would like to use Ray serve to set up an ecosystem of models behind a web app. We are having difficulty getting a deployment for local development.
When we try to use docker-compose with ray images we run into problems that the ray image python version (3.7) is not compatible with the web server’s version (3.9) which is needed for running fastAPI with all the pydantic sugar.
For local development we would like to be able to launch a few containers with ray running, e.g. a head and some workers, then deploy our fastAPI webserver on top of ray serve. This is our docker compose file:
services: ray-head: image: ray_head build: context: ./ dockerfile: ray_head/Dockerfile ports: - 8265:8265 - 8000:8000 volumes: - /dev/shm:/dev/shm networks: - safe-tier app: image: app build: context: ./ dockerfile: app/Dockerfile depends_on: - ray-head networks: - safe-tier networks: safe-tier:
where ray_head/Dockerfile entry point is:
ENTRYPOINT [ "ray", "start", "--head", "--port=6379", "--redis-shard-ports=6380,6381", "--object-manager-port=22345","--node-manager-port=22346","--dashboard-host=0.0.0.0","--block"]
where the app’s main entry point contains:
import ray from ray import serve from external_app import External if __name__ == '__main__': ray.init(address='ray://ray-head:10001') serve.start(detached=True) External.deploy()
This seems to launch the app, as it’s visible on the ray dashboard but then it’s not available at port 8000 on localhost despite being exposed to the host.
app-1 | (ServeController pid=256) INFO 2022-06-14 22:28:59,993 controller 256 deployment_state.py:1216 - Adding 1 replicas to deployment 'External'. app-1 | INFO:/usr/local/lib/python3.9/site-packages/ray/serve/client.py:Deployment 'External' is ready at `http://127.0.0.1:8000/api1`. component=serve deployment=External
How do we access the deployed app?