Ray job logs --follow does not cooperate with UNIX pipes

Monitoring & Debugging
1

How severe does this issue affect your experience of using Ray?

  • Medium

I am trying to flow ray job logs to a downstream UNIX pipeline. For example:

ray job logs --follow <jobid> | postprocessIt

This works fine if I don’t specify --follow. However, with --follow, the ray CLI seems to not behave like a good UNIX citizen. Nothing flows out of the pipe, if there is less than one “page” worth of data. Data is not streamed…

You can try this, which should work to stream both to the console,and to /tmp/foo.txt.

ray job logs <myjob> | tee /tmp/foo.txt

however, when you add a --follow, it does not seem to work.

ray --version
ray, version 1.12.1

Edit: I still see this in 1.13.0

Edit: also a problem with ray job logs -f <jobid> > /tmp/foo.txt, i.e. the tee part isn’t needed to repro this issue, it just saves you the step of having to separately do a tail -f /tmp/foo.txt

Workaround

For now, I will use websocat --no-line ws://.../api/jobs/<jobid>/logs/tail which cooperates with unix pipes. This however, means one needs to find a way to wait for job completion (which ray job logs -f does implicitly). See Feature request: CLI command `ray job wait`

2

cc: @architkulkarni Could you take a look?

3

Hi @starpit, sorry for the late reply, and thanks for sharing the details and the reproduction. Do you mind creating a Github issue for this? We can have someone look into it and track it there.