Can't access ray server outside docker container

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I am trying to set up run rayserve inside a docker container. The server is up and running inside the container but I can’t access the server at localhost:8000 outside the docker container. Based on my understanding, I need to specify a host of 0.0.0.0 when starting the ray server but I couldn’t find a way to do so despite spending hours trying suggested solutions from other threads. Could someone recommend the best way to achieve this with the latest ray image? Thank you!

Here is my docker-compose file:

services:
  ray-serve:
    container_name: ray-head
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - 8265:8265
      - 8000:8000
    privileged: true

And here is my Dockerfile:

# Dockerfile
FROM rayproject/ray:nightly-aarch64

# Install any additional packages
RUN pip install ray ray[serve] requests

WORKDIR /app
COPY deployments /app/deployments

ENTRYPOINT ["ray", "start", "--head", "--port=6379", "--dashboard-host=0.0.0.0","--block"]

Here is my server code which is under /deployments directory.

#serve.py
from ray import serve

from starlette.requests import Request
from typing import Dict

@serve.deployment
class Hello:
    def __init__(self):
        pass

    async def __call__(self, starlette_request: Request) -> Dict:
        req = await starlette_request.json()
        name = req["name"]
        return {"response": f"Hello {name}!"}

serve.run(Hello.bind())

I tried several approaches but none of them work:

  1. Using serve run serve:app --host 0.0.0.0 from this thread, I got an error: Error: No such option: --host
  2. I also tried serve.start(detached=True, http_options={"host": "0.0.0.0"} from this thread, but I couldn’t run Hello.deploy() in the end. I got an error saying Deployment object doesn’t have a method named deploy().
  3. I also tried serve deploy serve_config.yaml inside the container but I got an import error:
(base) ray@1ca327ef6946:/app/deployments$ serve status
applications:
  app1:
    status: DEPLOY_FAILED
    ...
      ModuleNotFoundError: No module named 'serve'
2

I figured it out after reading this thread. Bascially I need to add the following lines to the serve.py:

import ray

ray.init()
serve.start(detached=True, http_options={"host": "0.0.0.0"})

Then in the docker container, simply run: python serve.py to start serving.

The full serve.py file is as follows for future reference:

from ray import serve
from starlette.requests import Request
from typing import Dict
import ray

ray.init()
serve.start(detached=True, http_options={"host": "0.0.0.0"})

@serve.deployment
class Hello:
    def __init__(self):
        pass

    async def __call__(self, starlette_request: Request) -> Dict:
        req = await starlette_request.json()
        name = req["name"]
        return {"response": f"Hello {name}!"}

serve.run(Hello.bind())