ray 1.9 on windows 10
My program is hanging with Exception ignored in: ‘ray._raylet.task_execution_handler’ followed with a StackTrace Information of a list of PyInit__raylet’s. What should I be looking at to resolve the problem?
import ray
from ray import serve
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
http_options = {'host': "127.0.0.1", 'port': 8787, 'location': "HeadOnly", 'num_cpus': 2}
ray.init(address="127.0.0.1:8787", namespace="serve")
serve.start(http_options=http_options)
@serve.deployment(route_prefix="/api")
@serve.ingress(app)
class Deployment:
def __init__(self):
load_data()
@app.post("/first")
async def do_first(self):
etc.
etc.
C:\...\first.py
2022-02-02 18:46:26,018 INFO worker.py:842 -- Connecting to existing Ray cluster at address: 127.0.0.1:8787
2022-02-02 18:46:28,230 INFO api.py:414 -- Connecting to existing Serve instance in namespace 'serve'.
2022-02-02 18:46:28,280 INFO api.py:242 -- Updating deployment 'Deployment'. component=serve deployment=Deployment
pid=8712) 2022-02-02 18:46:28,380 INFO deployment_state.py:874 -- Stopping 1 replicas of deployment 'Deployment' with outdated versions. component=serve deployment=Deployment
pid=8712) 2022-02-02 18:46:48,699 INFO deployment_state.py:912 -- Adding 1 replicas to deployment 'Deployment'. component=serve deployment=Deployment
pid=13412) Exception ignored in: 'ray._raylet.task_execution_handler'
pid=13412) [2022-02-02 18:46:54,646 C 13412 12068] direct_actor_transport.cc:151: Check failed: objects_valid 0 1
pid=13412) *** StackTrace Information ***
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412) PyInit__raylet
pid=13412)
followed by multiple messages like this:
pid=9784) 2022-02-02 19:23:57,745 WARNING deployment_state.py:1123 -- Deployment 'Deployment' has 1 replicas that have taken more than 30s to initialize. This may be caused by a slow __init__ or reconfigure method.component=serve deployment=Deployment
pid=9784) 2022-02-02 19:24:27,747 WARNING deployment_state.py:1123 -- Deployment 'Deployment' has 1 replicas that have taken more than 30s to initialize. This may be caused by a slow __init__ or reconfigure method.component=serve deployment=Deployment