Workers not sending logs to driver soon enough. Can't use docker logs

Ray Core
1

Hi(repost from Slack),
I’m using Ray Serve(1.5.2) and running it in a docker container but this may be Ray Core question?

When I do docker logs -f <container_id> I find that some logs don’t show up (specifically in the _call_ method). It does show up eventually but I’m not sure when. For e.g. I have been waiting for 10 mins or more but I still don’t see logs for a request I sent in.

I have two deployments and it does show the info message for one deployment for e.g. which is in the __init__ method i.e. Ready to Serve

The “Start” and “End” messages haven’t shown up yet.

# util.py
def make_logger(name: str):
    _logger = logging.getLogger(name)
    # Added a streamhandler because of https://stackoverflow.com/a/63196863/238012
    fmt1 =  '%(asctime)s, %(levelname)-8s [%(filename)s:%(module)s:%(funcName)s:%(lineno)d] %(message)s'
    formatter = logging.Formatter(fmt1)
    ch = logging.StreamHandler()
    ch.setLevel(level=logging.INFO)
    ch.setFormatter(formatter)
    _logger.addHandler(ch)
    return _logger

#app.py
@serve.deployment(route_prefix="/predict", num_replicas=worker_count // 2)
class ImageModel:
    def __init__(self):
        self.logger = util.make_logger("ray")
        self.logger.info("Ready to Serve")

    async def __call__(self, starlette_request):
        self.logger.info("Start")
        print("Start")
        ..
        self.logger.info("End")
        print("End")

If I look at /tmp/ray/session_latest/logs, I can see my log entries, but just not in the driver logs (which I’m guessing is what docker logs hooks into)

Appreciate any help.

2

cc @simon-mo do you have context on this one?

3

Sorry for the late reply, these logs should be streamed to your driver scripts. Do you also start ray serve in the same script or it’s started before hand?

4

Hi Simon, thanks for getting back.

just to clarify, I don’t see all the logs in the driver logs for a while but much later(I don’t know when), it does show up. This is still an issue as I’m sometimes testing things out in the api and I don’t see logs or if something goes wrong in prod and I’m checking the logs, I don’t see the latest logs using docker logs <container_id>. We are working on Loki integration but I don’t know if that’ll help.

I start ray serve in the same script (app.py)

if __name__ == "__main__":

    ray.init(object_store_memory=500*10**6, ignore_reinit_error=True, dashboard_host="0.0.0.0")  # nosec

    model_actor = predictor_util.build_model_actor(MODEL_ACTOR, conf)
    serve.start(
        http_options={"host": "0.0.0.0"},  # nosec
    )
    ImageModel.deploy()


    while True:
        time.sleep(600)
5

if you don’t mind, can you share your logging setup in Ray Serve?

6

Hello, I am wondering if this was ever resolved as I appear to be experiencing the exact same issue.

I recently bumped from Ray v1.9.1 to Ray v1.13.0. After doing so, I stopped seeing logging statements in the docker logs. However, when I open up Ray Dashboard, the logging statements are there if I click on the Logs link.

If I revert to Ray v1.11.1, then the logging statements re-appear in the docker logs. The logging statements disappear again with Ray v1.12.1. I believe there was some change to logging in Ray Serve from v1.11 to v1.12, but I cannot figure out what the change would be and how to get the logging statements back in the docker logs.

I am using Ray Serve. I start a Ray cluster and Ray Serve from the CLI in an entrypoint script for the docker image. I am using Ray’s base image. The entrypoint script is:

ray start --head
serve start

Then, I have a python script that connects to the Ray cluster and Server:

ray.init(address="auto", namespace="serve")
Api.deploy()

while True:
    time.sleep(5)

Any information and/or guidance to get the logging statements to re-appear in Ray v1.12 or newer would be greatly appreciated.

7

I’m sorry, that project where I faced this has lost business priority so I stopped checking long time ago. It’s a hectic startup life so I won’t be able to go back and test it out :frowning:

8

@rabraham Thanks for the quick response and status update from your end. I appreciate it. I will start a new topic.

1 Like