# Workers not sending logs to driver soon enough. Can't use docker logs

**URL:** <https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497>\
**Category:** Ray Core\
**Created:** [September 10, 2021, 2:21pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497 "2021-09-10T14:21:13Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rabraham](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rabraham/32/532_2.png) [@rabraham](https://discuss.ray.io/u/rabraham)\
**Post date:** [September 10, 2021, 2:21pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/1 "2021-09-10T14:21:13Z")

</div>

Hi(repost from Slack),  
I’m using Ray Serve(1.5.2) and running it in a docker container but this may be Ray Core question?

When I do `docker logs -f <container_id>` I find that some logs don’t show up (specifically in the `_call_` method). It does show up eventually but I’m not sure when. For e.g. I have been waiting for 10 mins or more but I still don’t see logs for a request I sent in.

I have two deployments and it does show the info message for one deployment for e.g. which is in the ` __init__ ` method i.e. `Ready to Serve`

The “Start” and “End” messages haven’t shown up yet.

```auto
# util.py
def make_logger(name: str):
    _logger = logging.getLogger(name)
    # Added a streamhandler because of https://stackoverflow.com/a/63196863/238012
    fmt1 = '%(asctime)s, %(levelname)-8s [%(filename)s:%(module)s:%(funcName)s:%(lineno)d] %(message)s'
    formatter = logging.Formatter(fmt1)
    ch = logging.StreamHandler()
    ch.setLevel(level=logging.INFO)
    ch.setFormatter(formatter)
    _logger.addHandler(ch)
    return _logger

#app.py
@serve.deployment(route_prefix="/predict", num_replicas=worker_count // 2)
class ImageModel:
    def __init__ (self):
        self.logger = util.make_logger("ray")
        self.logger.info("Ready to Serve")

    async def __call__ (self, starlette_request):
        self.logger.info("Start")
        print("Start")
        ..
        self.logger.info("End")
        print("End")

```

If I look at `/tmp/ray/session_latest/logs`, I can see my log entries, but just not in the driver logs (which I’m guessing is what docker logs hooks into)

Appreciate any help.

---

<div class="post-metadata">

**Author:** ![Chen\_Shen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chen_shen/32/1486_2.png) [@Chen\_Shen](https://discuss.ray.io/u/Chen_Shen)\
**Post date:** [September 11, 2021, 11:13pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/2 "2021-09-11T23:13:37Z")

</div>

cc @simon-mo do you have context on this one?

---

<div class="post-metadata">

**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\
**Post date:** [September 17, 2021, 10:58pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/3 "2021-09-17T22:58:13Z")

</div>

Sorry for the late reply, these logs should be streamed to your driver scripts. Do you also start ray serve in the same script or it’s started before hand?

---

<div class="post-metadata">

**Author:** ![rabraham](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rabraham/32/532_2.png) [@rabraham](https://discuss.ray.io/u/rabraham)\
**Post date:** [September 18, 2021, 6:41pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/4 "2021-09-18T18:41:04Z")

</div>

Hi Simon, thanks for getting back.

just to clarify, I don’t see all the logs in the driver logs for a while but much later(I don’t know when), it does show up. This is still an issue as I’m sometimes testing things out in the api and I don’t see logs or if something goes wrong in prod and I’m checking the logs, I don’t see the latest logs using docker logs \<container\_id\>. We are working on Loki integration but I don’t know if that’ll help.

I start ray serve in the same script (`app.py`)

```auto
if __name__ == " __main__":

    ray.init(object_store_memory=500*10**6, ignore_reinit_error=True, dashboard_host="0.0.0.0") # nosec

    model_actor = predictor_util.build_model_actor(MODEL_ACTOR, conf)
    serve.start(
        http_options={"host": "0.0.0.0"}, # nosec
    )
    ImageModel.deploy()

    while True:
        time.sleep(600)

```

---

<div class="post-metadata">

**Author:** ![rabraham](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rabraham/32/532_2.png) [@rabraham](https://discuss.ray.io/u/rabraham)\
**Post date:** [September 21, 2021, 3:24pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/5 "2021-09-21T15:24:16Z")

</div>

if you don’t mind, can you share your logging setup in Ray Serve?

---

<div class="post-metadata">

**Author:** ![ts-cfield](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ts-cfield/32/2769_2.png) [@ts-cfield](https://discuss.ray.io/u/ts-cfield)\
**Post date:** [June 21, 2022, 3:10pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/6 "2022-06-21T15:10:38Z")

</div>

Hello, I am wondering if this was ever resolved as I appear to be experiencing the exact same issue.

I recently bumped from Ray v1.9.1 to Ray v1.13.0. After doing so, I stopped seeing logging statements in the docker logs. However, when I open up Ray Dashboard, the logging statements are there if I click on the Logs link.

If I revert to Ray v1.11.1, then the logging statements re-appear in the docker logs. The logging statements disappear again with Ray v1.12.1. I believe there was some change to logging in Ray Serve from v1.11 to v1.12, but I cannot figure out what the change would be and how to get the logging statements back in the docker logs.

I am using Ray Serve. I start a Ray cluster and Ray Serve from the CLI in an entrypoint script for the docker image. I am using Ray’s base image. The entrypoint script is:

```nohighlight
ray start --head
serve start

```

Then, I have a python script that connects to the Ray cluster and Server:

```python
ray.init(address="auto", namespace="serve")
Api.deploy()

while True:
    time.sleep(5)

```

Any information and/or guidance to get the logging statements to re-appear in Ray v1.12 or newer would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![rabraham](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rabraham/32/532_2.png) [@rabraham](https://discuss.ray.io/u/rabraham)\
**Post date:** [June 22, 2022, 10:01am UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/7 "2022-06-22T10:01:07Z")

</div>

I’m sorry, that project where I faced this has lost business priority so I stopped checking long time ago. It’s a hectic startup life so I won’t be able to go back and test it out ☹

---

<div class="post-metadata">

**Author:** ![ts-cfield](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ts-cfield/32/2769_2.png) [@ts-cfield](https://discuss.ray.io/u/ts-cfield)\
**Post date:** [June 22, 2022, 1:07pm UTC](https://discuss.ray.io/t/workers-not-sending-logs-to-driver-soon-enough-cant-use-docker-logs/3497/8 "2022-06-22T13:07:15Z")

</div>

@rabraham Thanks for the quick response and status update from your end. I appreciate it. I will start a new topic.