Hi All,
I am studying mlops and i’m trying to use ray serve to serving model but after few time running success, this error appear and idk what happen @@!
(dtungvu) dtungvu@aiteam-960XFH:/srv/scratch/dtung.vu/Exp/RayServe/Exp01$ serve run model:detector
2024-01-18 15:02:23,009 INFO scripts.py:438 -- Running import path: 'model:detector'.
2024-01-18 15:02:25,596 INFO worker.py:1715 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
(ProxyActor pid=226460) INFO 2024-01-18 15:02:27,260 proxy 107.98.152.63 proxy.py:1128 - Proxy actor e36b05810866aa5d54697fe101000000 starting on node b383adeb90de4b70dbfa55ee8adbe5d4c9a69d59cf4c1d80fdfd939e.
(ProxyActor pid=226460) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-b383adeb90de4b70dbfa55ee8adbe5d4c9a69d59cf4c1d80fdfd939e:ProxyActor.__init__() (pid=226460, ip=107.98.152.63, actor_id=e36b05810866aa5d54697fe101000000, repr=<ray.serve._private.proxy.ProxyActor object at 0x7f3db89a9370>)
(ProxyActor pid=226460) File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/concurrent/futures/_base.py", line 446, in result
(ProxyActor pid=226460) return self.__get_result()
(ProxyActor pid=226460) File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
(ProxyActor pid=226460) raise self._exception
(ProxyActor pid=226460) File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/serve/_private/proxy.py", line 1202, in __init__
(ProxyActor pid=226460) self.wrapped_http_proxy, **middleware.options
(ProxyActor pid=226460) AttributeError: 'Middleware' object has no attribute 'options'
Traceback (most recent call last):
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/bin/serve", line 8, in <module>
sys.exit(cli())
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/serve/scripts.py", line 472, in run
client = _private_api.serve_start(
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/serve/_private/api.py", line 291, in serve_start
controller, controller_name = _start_controller(
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/serve/_private/api.py", line 175, in _start_controller
ray.get(
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/aiteam/jupyterhub/dtungvu/.conda/envs/dtungvu/lib/python3.9/site-packages/ray/_private/worker.py", line 2626, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: ProxyActor
actor_id: e36b05810866aa5d54697fe101000000
pid: 226460
name: SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-b383adeb90de4b70dbfa55ee8adbe5d4c9a69d59cf4c1d80fdfd939e
namespace: serve
ip: 107.98.152.63
The actor is dead because it was killed by `ray.kill`.
(ServeController pid=226405) WARNING 2024-01-18 15:02:31,704 controller 226405 proxy_state.py:537 - Didn't receive ready check response for proxy b383adeb90de4b70dbfa55ee8adbe5d4c9a69d59cf4c1d80fdfd939e after 5.0s.
(ServeController pid=226405) INFO 2024-01-18 15:02:31,705 controller 226405 proxy_state.py:805 - Proxy on node 'b383adeb90de4b70dbfa55ee8adbe5d4c9a69d59cf4c1d80fdfd939e' UNHEALTHY. Shutting down the unhealthy proxy and starting a new one.
Please give me some help !! Tks !!1