FastAPI and Websockets

Ray Serve
1

So I really enjoy the simple Ray FastAPI deployment, however I am unable to add websockets to my app. Is this a missing feature or am I simply doing something wrong?

1 Like
2

Thanks for the question! Currently (2021/10) Serve doesn’t support websockets in FastAPI integration. You are not doing anything wrong. Can you explain a bit about the use case? Are you using websockets to stream predictions?

3

Hi Simon, thank you for your reply.

The use case is a machine learning application presenting results (opencv images) in a videostream. The video is currently streaming over ZeroMQ. However I was thinking I could reduce one dependency if I could stream over websocket.

4

Is there any progress or any change in this regard?
Or is there any recommendation about how to connect a websocket server with ray deployments?

We did a quite simple test:

@serve.deployment(
    name="websocket-test",
    version='v1',
    route_prefix="/test",
    ray_actor_options={
        'num_cpus': 0
    }
)
@serve.ingress(app)
class WebsocketServer:

    def __init__(self):
        self._uuid = str(uuid.uuid4())

    @app.get('/')
    async def get(self):
        return HTMLResponse(html)

    @app.websocket("/ws")
    async def websocket_endpoint(self, websocket: WebSocket):
        await websocket.accept()
        while True:
            data = await websocket.receive_text()
            await websocket.send_text(f"Message text was: {data}")

And had following error:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/juan/miniconda3/envs/dala-37/lib/python3.7/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 199, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
  File "/home/juan/miniconda3/envs/dala-37/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/juan/miniconda3/envs/dala-37/lib/python3.7/site-packages/ray/serve/http_proxy.py", line 269, in __call__
    assert scope["type"] == "http"
AssertionError
INFO:     connection open
INFO:     connection closed
5

Please open a feature request here and We can prioritize it in our roadmap.

6

Done: [Serve] Allow WebSocket endpoints for ray deployments using FastAPI · Issue #25363 · ray-project/ray · GitHub

Thanks for your time

7

Did it got updated? can we do inference in ray serve using web socket connection?