Separate log for each actor task

Dashboard, Monitoring & Debugging
1

Hi,

I couldn’t find anywhere in the docs a way to get logs isolated for each task in the same Actor. For example we have a threaded Actor with many tasks running at the same time, and in the dashboard it would be great that when we clicked on the log for a task (thread in this case) it would show only the logs for that task and not all logs from that Actor.

I figure there should be a way since the task has an ID in the dashboard and this id is also available to each task (ray.get_runtime_context().get_task_id()). But I can’t figure out how the dashboard turns the worker logs that I can find on the machine into the log displayed in the dashboard for each task.

Any help appreciated,
Thanks!

2

Here is my understanding. Need @aguo to confirm:

It’s a bit challenging for Ray to achieve this since there is no log query backend in a Ray cluster. Ray dashboard reads log files directly. There might be hacky ways to make it work for Ray dashboard, like add some specials identifiers in the log file for each task but it’s not a robust solution.

The best way to do it is probably define your own logger, enable structured logs, and ingest the log to another logging product like Loki, cloud watch, etc. then u can query the task log directly.

You can also try Anyscale since it also adds log query feature for this type of use case and it can better integrate log query into a customized ray dashboard

3

From experimenting it doesn’t seem like a direct read. When looking at the log file for the only worker, it has all logs output on that worker, but then the actor tasks only have the logs from within the Actor. And tasks that ended earlier only have some of the logs, while the longest task has the logs from everything

4

Hm. I’m not exactly sure how this works then.

@alan or someone on core team should @jjyao @eoakes should confirm

5

any updates? We would love to stick with just Ray if possible

6

Each worker writes logs to a log file. There is two ways log lines are distinguished between tasks.

One is a magic token that gets printed to the log file at the beginning and end of each task. These tokens are then parsed by the Ray Dashboard. This magic token technique does not work well with threaded actors because log lines can be interspersed with each other.

The second way is to enable structured logging and then parsing and filtering the log files using external tools: Configuring Logging — Ray 2.45.0