Workers not sending logs to driver soon enough. Can't use docker logs

Hi(repost from Slack),
I’m using Ray Serve(1.5.2) and running it in a docker container but this may be Ray Core question?

When I do docker logs -f <container_id> I find that some logs don’t show up (specifically in the _call_ method). It does show up eventually but I’m not sure when. For e.g. I have been waiting for 10 mins or more but I still don’t see logs for a request I sent in.

I have two deployments and it does show the info message for one deployment for e.g. which is in the __init__ method i.e. Ready to Serve

The “Start” and “End” messages haven’t shown up yet.

def make_logger(name: str):
    _logger = logging.getLogger(name)
    # Added a streamhandler because of
    fmt1 =  '%(asctime)s, %(levelname)-8s [%(filename)s:%(module)s:%(funcName)s:%(lineno)d] %(message)s'
    formatter = logging.Formatter(fmt1)
    ch = logging.StreamHandler()
    return _logger
@serve.deployment(route_prefix="/predict", num_replicas=worker_count // 2)
class ImageModel:
    def __init__(self):
        self.logger = util.make_logger("ray")"Ready to Serve")

    async def __call__(self, starlette_request):"Start")

If I look at /tmp/ray/session_latest/logs, I can see my log entries, but just not in the driver logs (which I’m guessing is what docker logs hooks into)

Appreciate any help.

cc @simon-mo do you have context on this one?

Sorry for the late reply, these logs should be streamed to your driver scripts. Do you also start ray serve in the same script or it’s started before hand?

Hi Simon, thanks for getting back.

just to clarify, I don’t see all the logs in the driver logs for a while but much later(I don’t know when), it does show up. This is still an issue as I’m sometimes testing things out in the api and I don’t see logs or if something goes wrong in prod and I’m checking the logs, I don’t see the latest logs using docker logs <container_id>. We are working on Loki integration but I don’t know if that’ll help.

I start ray serve in the same script (

if __name__ == "__main__":

    ray.init(object_store_memory=500*10**6, ignore_reinit_error=True, dashboard_host="")  # nosec

    model_actor = predictor_util.build_model_actor(MODEL_ACTOR, conf)
        http_options={"host": ""},  # nosec

    while True:

if you don’t mind, can you share your logging setup in Ray Serve?

Hello, I am wondering if this was ever resolved as I appear to be experiencing the exact same issue.

I recently bumped from Ray v1.9.1 to Ray v1.13.0. After doing so, I stopped seeing logging statements in the docker logs. However, when I open up Ray Dashboard, the logging statements are there if I click on the Logs link.

If I revert to Ray v1.11.1, then the logging statements re-appear in the docker logs. The logging statements disappear again with Ray v1.12.1. I believe there was some change to logging in Ray Serve from v1.11 to v1.12, but I cannot figure out what the change would be and how to get the logging statements back in the docker logs.

I am using Ray Serve. I start a Ray cluster and Ray Serve from the CLI in an entrypoint script for the docker image. I am using Ray’s base image. The entrypoint script is:

ray start --head
serve start

Then, I have a python script that connects to the Ray cluster and Server:

ray.init(address="auto", namespace="serve")

while True:

Any information and/or guidance to get the logging statements to re-appear in Ray v1.12 or newer would be greatly appreciated.

I’m sorry, that project where I faced this has lost business priority so I stopped checking long time ago. It’s a hectic startup life so I won’t be able to go back and test it out :frowning:

@rabraham Thanks for the quick response and status update from your end. I appreciate it. I will start a new topic.

1 Like