Workers not sending logs to driver soon enough. Can't use docker logs

Hi(repost from Slack),
I’m using Ray Serve(1.5.2) and running it in a docker container but this may be Ray Core question?

When I do docker logs -f <container_id> I find that some logs don’t show up (specifically in the _call_ method). It does show up eventually but I’m not sure when. For e.g. I have been waiting for 10 mins or more but I still don’t see logs for a request I sent in.

I have two deployments and it does show the info message for one deployment for e.g. which is in the __init__ method i.e. Ready to Serve

The “Start” and “End” messages haven’t shown up yet.

# util.py
def make_logger(name: str):
    _logger = logging.getLogger(name)
    # Added a streamhandler because of https://stackoverflow.com/a/63196863/238012
    fmt1 =  '%(asctime)s, %(levelname)-8s [%(filename)s:%(module)s:%(funcName)s:%(lineno)d] %(message)s'
    formatter = logging.Formatter(fmt1)
    ch = logging.StreamHandler()
    ch.setLevel(level=logging.INFO)
    ch.setFormatter(formatter)
    _logger.addHandler(ch)
    return _logger

#app.py
@serve.deployment(route_prefix="/predict", num_replicas=worker_count // 2)
class ImageModel:
    def __init__(self):
        self.logger = util.make_logger("ray")
        self.logger.info("Ready to Serve")

    async def __call__(self, starlette_request):
        self.logger.info("Start")
        print("Start")
        ..
        self.logger.info("End")
        print("End")

If I look at /tmp/ray/session_latest/logs, I can see my log entries, but just not in the driver logs (which I’m guessing is what docker logs hooks into)

Appreciate any help.

cc @simon-mo do you have context on this one?

Sorry for the late reply, these logs should be streamed to your driver scripts. Do you also start ray serve in the same script or it’s started before hand?

Hi Simon, thanks for getting back.

just to clarify, I don’t see all the logs in the driver logs for a while but much later(I don’t know when), it does show up. This is still an issue as I’m sometimes testing things out in the api and I don’t see logs or if something goes wrong in prod and I’m checking the logs, I don’t see the latest logs using docker logs <container_id>. We are working on Loki integration but I don’t know if that’ll help.

I start ray serve in the same script (app.py)

if __name__ == "__main__":

    ray.init(object_store_memory=500*10**6, ignore_reinit_error=True, dashboard_host="0.0.0.0")  # nosec

    model_actor = predictor_util.build_model_actor(MODEL_ACTOR, conf)
    serve.start(
        http_options={"host": "0.0.0.0"},  # nosec
    )
    ImageModel.deploy()


    while True:
        time.sleep(600)