Workers not sending logs to driver soon enough. Can't use docker logs

Ray Core
6

Hello, I am wondering if this was ever resolved as I appear to be experiencing the exact same issue.

I recently bumped from Ray v1.9.1 to Ray v1.13.0. After doing so, I stopped seeing logging statements in the docker logs. However, when I open up Ray Dashboard, the logging statements are there if I click on the Logs link.

If I revert to Ray v1.11.1, then the logging statements re-appear in the docker logs. The logging statements disappear again with Ray v1.12.1. I believe there was some change to logging in Ray Serve from v1.11 to v1.12, but I cannot figure out what the change would be and how to get the logging statements back in the docker logs.

I am using Ray Serve. I start a Ray cluster and Ray Serve from the CLI in an entrypoint script for the docker image. I am using Ray’s base image. The entrypoint script is:

ray start --head
serve start

Then, I have a python script that connects to the Ray cluster and Server:

ray.init(address="auto", namespace="serve")
Api.deploy()

while True:
    time.sleep(5)

Any information and/or guidance to get the logging statements to re-appear in Ray v1.12 or newer would be greatly appreciated.