How to access my internal worker logs at one place

Monitoring & Debugging
1

I have my internal logging , how i can able to see those logs at worker level in ray dashboard ?

Each worker will have it’s own log file . it is possible to stream those logs somewhere ?

2

Yup - see here for how to configure application logging: Adding Application-Level Metrics — Ray 2.23.0

Prometheus + Grafana are good options if you haven’t selected a observability tool already. See here on how to set that up: Configuring and Managing Ray Dashboard — Ray 2.23.0

3

Hey
Thanks for the response.
The answer you have given will work , but my expectation was i have internal log files in a specific directory as .log file is it possible for ray to understand to read and stream it back as logs of the nodes ?

some sort of configuration can i add in ray init ??

4

Like you want a card in the Ray Dashboard that streams the incoming logs for all executing tasks per node?

That we don’t have as part of the Dashboard; hence the recommendation to use Prometheus instead.

1 Like
5

One thing try:

  • You can try configure the logger to save those log files in the ray logs folder (/tmp/ray/session_*/logs). Then they might show up in Ray Dashboard.

Not sure how you can retrieve the session id and set it up dynamically though.

6

Thanks for the replay , even i had Prometheus as last option .