How to retrieve a dead node logs

When one ray node fails, I’d like to access the logs, but neither from dashboard nor state cli/api, I can figure out how to retrieve the logs.

ray.experimental.state.exception.RayStateApiException: API server internal error. See dashboard.log file for more details. Error: Given node id 9a284b2859493946ad23e4ba1b62b0343380fac82d6b492a850f88bb is not available. It's either the node is dead, or it is not registered. Use `ray list nodes` to see the node status. If the node is registered, it is highly likely a transient issue. Try again.

cc: @sangcho @rickyyx
I guess ray has no way to access logs after node is dead… you probably need to persist the logs themselves somehow.

This is not supported, and we don’t plan to support it from ray dashboard. If it is needed, please persist logs from your infrastructure (setting a log agent and persist log files). If you use kuberay, you can also check Logging — Ray 3.0.0.dev0.

1 Like