Check if ingress class init function has finished running on Ray Serve

Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I’m using Ray Serve to deploy my OCR model as a service. This is the relevant portion of my code:

fastapi_ocr_app = FastAPI()

@serve.deployment
@serve.ingress(fastapi_ocr_app)
class OCRIngress:
    def __init__(self) -> None:
        self.engine = OCREngine(x="a", y="b")
...

The __init__ function here takes a while to finish running since it may have to download some artifacts depending on what the values of x and y are. Now in my CI pipeline, I deploy this as a service with a docker compose up -d command and then later down the line I run some integration tests to make sure everything is functional.

Now the problem is that I need a way to ensure that the __init__ function has actually finished running so that when I’m running my tests I don’t get false negatives due to the replica not having finished initializing.

Is there a way to check for this specific problem? To reiterate, I need a way of checking if the __init__ function of an ingress class has finished executing so that I can run tests safely without running into issues due to the class not having been properly initialized. I would also rather not do a sleep X prior to the tests because the downloads are cached so the next runs of the same CI pipeline don’t actually need to wait anymore.

Thanks in advance for your help!

2

I can think of a couple options:

  1. You can poll serve status and wait for the app to enter a RUNNING state and every deployment to enter a HEALTHY state.
  2. If you’re only starting 1 replica, you can submit a request and wait until the request returns successfully.