Hi(repost from Slack),
I’m using Ray Serve(1.5.2) and running it in a docker container but this may be Ray Core question?
When I do
docker logs -f <container_id> I find that some logs don’t show up (specifically in the
_call_ method). It does show up eventually but I’m not sure when. For e.g. I have been waiting for 10 mins or more but I still don’t see logs for a request I sent in.
I have two deployments and it does show the info message for one deployment for e.g. which is in the
__init__ method i.e.
Ready to Serve
The “Start” and “End” messages haven’t shown up yet.
# util.py def make_logger(name: str): _logger = logging.getLogger(name) # Added a streamhandler because of https://stackoverflow.com/a/63196863/238012 fmt1 = '%(asctime)s, %(levelname)-8s [%(filename)s:%(module)s:%(funcName)s:%(lineno)d] %(message)s' formatter = logging.Formatter(fmt1) ch = logging.StreamHandler() ch.setLevel(level=logging.INFO) ch.setFormatter(formatter) _logger.addHandler(ch) return _logger #app.py @serve.deployment(route_prefix="/predict", num_replicas=worker_count // 2) class ImageModel: def __init__(self): self.logger = util.make_logger("ray") self.logger.info("Ready to Serve") async def __call__(self, starlette_request): self.logger.info("Start") print("Start") .. self.logger.info("End") print("End")
If I look at
/tmp/ray/session_latest/logs, I can see my log entries, but just not in the driver logs (which I’m guessing is what docker logs hooks into)
Appreciate any help.