Routing with DAGDriver and serve.ingress

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I have:

@serve.deployment
@serve.ingress(app)
class Hello:

    @app.get("/")
    async def hello(request: Request) -> str:
        return "hello"
d = DAGDriver.bind({"/hello": Hello.bind()})

which deploys fine but then:

curl  -v localhost:8000/hello 
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /hello HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Sun, 25 Sep 2022 17:30:22 GMT
< server: uvicorn
< content-length: 196
< content-type: application/json
< 
* Connection #0 to host localhost left intact
{"messages":[{"type":"http.response.start","status":404,"headers":[["content-length","22"],["content-type","application/json"]]},{"type":"http.response.body","body":"{\"detail\":\"Not Found\"}"}]}

which is confusing because it’s giving a 200, but a body saying not found

2

Hi @PaulRudin , Thank you for raisin questions! please try

@serve.deployment
@serve.ingress(app)
class Hello:

    @app.get("/hello")
    async def hello(request) -> str:
        return "hello"
serve.run(Hello.bind())

We are improving our error message & documentation in the future releases!

3

Thanks. I understand that works, but I specifically am interested with the case with both DAGDriver and @serve.ingress, because I want to be able to use FastApi wildcard routes, but also have some deployments that can be scaled independently.

The question is really what specifically goes wrong with my example - why don’t we just get a 200 with “hello” as the response body?

4

Intentionally DAGDriver and Ingress are not encouraged to be used together. Inside the DAGDriver, we are having an internal FastAPI to support http traffic routing (which is redundant with your FastAPI). If you use them together, it will have two FastAPI apps, one is nested into the other.
To make it hacky working, you can change to:

@serve.deployment
@serve.ingress(app)
class Hello:

    @app.get("/hello")
    async def hello(request) -> str:
        print("in hello")
        return "hello"
serve.run(DAGDriver.bind({"/hello": Hello.bind()}))
5

Hmm - that’s a bit counterintuitive - since you’d expect the route to be /hello/hello in that case.

I guess that this really means if you want to use fastapi wildcard routes you need to put all your routes through one fastapi deployment? Which I suppose is OK, but you lose the possibility of independently scalable deployments.

I suppose more generally if there’s a fastapi instance under the hood in any case, it would be nice to have support for wildcarded routes without using @serve.ingress(app).

1 Like