Redirecting All Ray Logging to a single file

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi

Is there a way to redirect all Ray logs (from workers, actors, monitoring, dashboard, etc.) to a single file defined by the user?
I found this reference which shows how to redirect the logs to stderr but I was wondering if I can provide a custom filename instead: Logging — Ray 2.3.0

Also, what if I want to redirect the logs from other nodes to a single file on the head node? Is there a way to accomplish this goal?

Finally, if the user can redirect the logs to a single file, does that mean the logging directory created under /tmp/ directory will not be created? In my environment there are restrictions on writing to disk, so I would prefer to avoid writing to disk space as much as possible

2

@amohar as far as I know this is not doable today.
cc @sangcho @rickyyx Do you know if we allow users to configure the logs, or is it something on the road map?

3

Hi Chen,

Thanks for the response, regarding my last question, so just to confirm, there is no way to avoid writing logs to disk on all nodes? In my environment there are restrictions on writing to disk, so I would prefer to avoid writing to disk space as much as possible

4

I guess this is based on the premise that compute is ephemeral i.e. it won’t go away like EC2 where the storage is lost with the node.

I am curious to know how can one consume one log file and find issues. I personally, find it somewhat overwhelming to jump through 3-5 log files.

As a last resort, can a bash script be run to move the logs to a centralized location which in Cloud would most likely be an S3 like storage.

5

@amohar

there is no way to avoid writing logs to disk on all nodes

One possibility is using a in-memory fs and change log dir to it.