Can't deploy models using Ray Serve

Ray Libraries (Data, Train, Tune, Serve)
1

I am trying to run the example given in ray documentation Serving a Distilbert Model — Ray 2.9.0.

While following the instructions I am getting the below error and ray is killing the service.

I am running ray in a python virtual environment.

I might be missing something here, can anyone help in figuring out what need to be done here?

2024-01-19 11:10:21,488	INFO scripts.py:438 -- Running import path: 'test_serve:app'.
2024-01-19 11:10:25,184	INFO worker.py:1715 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 
(ProxyActor pid=41279) INFO 2024-01-19 11:10:31,597 proxy 172.16.11.225 proxy.py:1141 - Proxy actor 56f5ac4526e76388e258b67101000000 starting on node 0996f9146a6b15ba1d323e7542da2d93b5b8791f51203149a89ccb45.
(ProxyActor pid=41279) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-0996f9146a6b15ba1d323e7542da2d93b5b8791f51203149a89ccb45:ProxyActor.__init__() (pid=41279, ip=172.16.11.225, actor_id=56f5ac4526e76388e258b67101000000, repr=<ray.serve._private.proxy.ProxyActor object at 0x7fb6443768b0>)
(ProxyActor pid=41279)   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
(ProxyActor pid=41279)     return self.__get_result()
(ProxyActor pid=41279)   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
(ProxyActor pid=41279)     raise self._exception
(ProxyActor pid=41279)   File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/serve/_private/proxy.py", line 1215, in __init__
(ProxyActor pid=41279)     self.wrapped_http_proxy, **middleware.options
(ProxyActor pid=41279) AttributeError: 'Middleware' object has no attribute 'options'
Traceback (most recent call last):
  File "/home/foundation/virtualenvs/ray/bin/serve", line 8, in <module>
    sys.exit(cli())
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/serve/scripts.py", line 472, in run
    client = _private_api.serve_start(
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/serve/_private/api.py", line 291, in serve_start
    controller, controller_name = _start_controller(
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/serve/_private/api.py", line 175, in _start_controller
    ray.get(
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/home/foundation/virtualenvs/ray/lib/python3.8/site-packages/ray/_private/worker.py", line 2626, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
	class_name: ProxyActor
	actor_id: 56f5ac4526e76388e258b67101000000
	pid: 41279
	name: SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-0996f9146a6b15ba1d323e7542da2d93b5b8791f51203149a89ccb45
	namespace: serve
	ip: 172.16.11.225
The actor is dead because it was killed by `ray.kill`.
(ServeController pid=41196) WARNING 2024-01-19 11:10:34,048 controller 41196 proxy_state.py:537 - Didn't receive ready check response for proxy 0996f9146a6b15ba1d323e7542da2d93b5b8791f51203149a89ccb45 after 5.0s.
(ServeController pid=41196) INFO 2024-01-19 11:10:34,048 controller 41196 proxy_state.py:805 - Proxy on node '0996f9146a6b15ba1d323e7542da2d93b5b8791f51203149a89ccb45' UNHEALTHY. Shutting down the unhealthy proxy and starting a new one.```
2

Same problem here. I believe it’s due to some changes in the latest fastapi or starlette. The temporary solution is to install fastapi==0.108.0

1 Like