# Redirect worker logs to the driver

**URL:** <https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136>\
**Category:** Dashboard, Monitoring & Debugging\
**Created:** [April 7, 2023, 9:15pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136 "2023-04-07T21:15:02Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![felixy12](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/felixy12/32/4197_2.png) [@felixy12](https://discuss.ray.io/u/felixy12)\
**Post date:** [April 7, 2023, 9:15pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/1 "2023-04-07T21:15:03Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Medium: It contributes to significant difficulty to complete my task, but I can work around it.

When a task/actor is executed, the `stdout/stderr` is redirected to the driver as well as to worker logs. Would it be possible to directly also forward those worker logs to the driver, for example in the driver’s `/tmp` directory? This would allow easy access to the per-worker logs that aren’t tangled into a single pipe.

Currently, I have a context manager that pipes all `stdout/stderr` of my driver and parses the outputs into files based on PID. While this works, there are some limitations (the blocking `.get()` call must happen within the context manager). It would be great if the above were possible to avoid this.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 8, 2023, 6:24am UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/2 "2023-04-08T06:24:57Z")

</div>

Not sure if I understand the question

> When a task/actor is executed, the `stdout/stderr` is redirected to the driver as well as to worker logs. Would it be possible to directly also forward those worker logs to the driver, for example in the driver’s `/tmp` directory?

What’s the difference between what ray does now and “directly also forward those worker logs to the driver”? (edited)

Is it about the the prefixed pid, etc.?

---

<div class="post-metadata">

**Author:** ![felixy12](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/felixy12/32/4197_2.png) [@felixy12](https://discuss.ray.io/u/felixy12)\
**Post date:** [April 9, 2023, 3:50pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/3 "2023-04-09T15:50:16Z")

</div>

Hi, thank you for the quick response! The difference with current behavior would be to have the logs in files directly, with different workers writing to different files.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 10, 2023, 4:27am UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/4 "2023-04-10T04:27:42Z")

</div>

I’m not sure if I really understand your question. Let me try to clarify a little bit.

- If you are familiar with the [Ray’s logging structure](https://docs.ray.io/en/latest/ray-observability/ray-logging.html#logging-directory-structure), Ray writes worker logs to files to the disk of the nodes where the workers (tasks and actors) are run.  
 ![Screen Shot 2023-04-09 at 9.12.56 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/0/09061dda86a1119c638684180e6d5c800f28455a.png)
- At the same time, if you enable `log_to_driver`, worker logs are redirected to the driver output.

Does it make sense? Can you rephrase your questions?

---

<div class="post-metadata">

**Author:** ![felixy12](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/felixy12/32/4197_2.png) [@felixy12](https://discuss.ray.io/u/felixy12)\
**Post date:** [April 10, 2023, 12:44pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/5 "2023-04-10T12:44:49Z")

</div>

Yep, that makes sense. My understanding is that the current behavior is that worker logs are:

- Written to log FILES on the worker nodes.
- Redirected to `stderr/stdout` on the driver.

My question is: Is there a straightforward way to have the log FILES that are written on the worker nodes to be written to the driver? I wouldn’t just want the logs to be redirected to driver output, but to be written into files onto the driver itself. Each worker node’s logs would correspond to a different file on the driver.

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 10, 2023, 4:31pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/6 "2023-04-10T16:31:53Z")

</div>

> have the log FILES that are written on the worker nodes to be written to the driver

You mean to write the worker log files to the head node? Driver is just the entrypoint script.

> Each worker node’s logs would correspond to a different file on the driver.

Each worker node may have a lot of worker log files. Do you just mean sending those log files to the head node? Or do you want to merge the worker log files first into 1 log file and send it to the head node then?

---

<div class="post-metadata">

**Author:** ![felixy12](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/felixy12/32/4197_2.png) [@felixy12](https://discuss.ray.io/u/felixy12)\
**Post date:** [April 11, 2023, 2:33pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/7 "2023-04-11T14:33:42Z")

</div>

This discussion may benefit from me describing the root issue, rather than focus on the specific technical details.

The general issue I’m running into is the following: On my AWS instance, I connect to a Ray server, and I send ~20 tasks to be done remotely on the server, using `ray.remote`. I would like to see the outputs form those tasks, but currently run into the following blockers:

- The stderr/stdout forwarded from the workers to my instance are jumbled together and difficult to parse.
- There is no easy way to know which worker’s log file I should check to find the outputs.
- These log files don’t persist after the task is done, as far as I can tell.

I would like to easily find the logs associated with each of my individual tasks, and for those log files to persist when the task completes.

Do you have any suggestions?

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [April 11, 2023, 5:17pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/8 "2023-04-11T17:17:33Z")

</div>

cc: @sangcho to follow up since he is working on the improvements to ray logging.

---

<div class="post-metadata">

**Author:** ![felixy12](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/felixy12/32/4197_2.png) [@felixy12](https://discuss.ray.io/u/felixy12)\
**Post date:** [April 17, 2023, 12:41pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/9 "2023-04-17T12:41:07Z")

</div>

Hi! Circling back on this. Any news?

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [April 17, 2023, 1:38pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/10 "2023-04-17T13:38:07Z")

</div>

Sorry. Just saw the thread.’

Btw, for one clarification; We don’t technically redirect logs to the driver. What we do is just to read a log file and stream those into the driver using RPC, and then driver just prints them again.

> The stderr/stdout forwarded from the workers to my instance are jumbled together and difficult to parse.

Is it because the output is always formatted in a random way like (name, pid=XYZ)?

> There is no easy way to know which worker’s log file I should check to find the outputs.

This is actually a problem we are planning to tackle in this version… For this one, I have a couple questions.

1. How do you plan to associate the worker logs to the job? via job\_id?
2. Do you use any logging tools & agents? For example, something like this [Log Persistence — Ray 3.0.0.dev0](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/logging.html)
3. Can you tell me a bit more about why just storing the driver output (which contains all worker logs technically) is not enough?

---

<div class="post-metadata">

**Author:** ![felixy12](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/felixy12/32/4197_2.png) [@felixy12](https://discuss.ray.io/u/felixy12)\
**Post date:** [April 17, 2023, 2:13pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/11 "2023-04-17T14:13:29Z")

</div>

Thanks for getting back to me! To try and address your questions,

> Is it because the output is always formatted in a random way like (name, pid=XYZ)?

Sort of? I want to use the logs to track progress of my tasks. It’s hard to track progress of 20 jobs when the logs are all funneled into a single stream. Although having clearer identification of where each log comes from would help, just that alone doesn’t solve the underlying issue.  
To create a dummy example, say I had a function that printed 0 to 100, taking some random amount of time in between each print. I launch 20 tasks that each run this function separately.

```auto
def print100():
    for i in range(100):
        print(i)
        time.sleep(random.random())

ray.init()
for _ in range(20):
    print100.remote()

```

I’d like to track the progress of each task. If the `stderr/stdout` of each task was re-directed to a separate file, this is easy. I can just look at each of the files and see which numbers have been printed for each. However, when all of prints are directed into a single stream, this is no longer easy to parse by eye.

For your latter series of questions:

1. I don’t know. Currently, calling `.remote()` does not return anything that can be used to associate worker logs/files with it. It would be great if that were the case.
2. I’m unfamiliar with the link that you sent, but can give it a look.
3. I think that is addressed in my example above, but let me know if that’s not the case.

Thanks!

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [May 8, 2023, 5:53pm UTC](https://discuss.ray.io/t/redirect-worker-logs-to-the-driver/10136/12 "2023-05-08T17:53:08Z")

</div>

![Screen Shot 2023-05-08 at 10.51.57 AM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/4/4c8b3dbbfb38a7898ae123bb3e5adc619e348591.png)

@felixy12 It seems that you just want to see the logs of each of your ray tasks.  
Have you tried to look at those logs via ray dashboard? The can click the log link in the dashboard.