[Client] [Clusters] How to persist the entire driver log while submitting jobs with `tmux` / `screen`

Ray Core Ray Client
1

Hello!

What would be a way to persist the entire driver log while submitting a job with something like ray submit --tmux ray_cluster.yaml --cluster-name jkkwon ?

Simply redirecting with > logs.txt doesn’t work, since that’ll only contain lines like below

Loaded cached provider configuration
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
Fetched IP: 54.187.23.170
Fetched IP: 54.187.23.170
Run `ray attach ray_cluster.yaml --cluster-name=jkkwon--tmux` to check command status.

When I do attach to the cluster (ray attach ray_cluster.yaml --cluster-name=jkkwon--tmux), I just see the log from tailing, not the entire log.

@sangcho

2

cc @ijrsvt @simon-mo Do you guys know what’s the best way to achieve it?

3

submit probably won’t do it because it just runs python filename.py in the head node. Maybe try this?

  • First sync the files over ray rsync-up cluster.yaml
  • Then exec the script in the back ground with log redirection ray exec --tmux "python driver.py > logs.txt"

Regular shell redirection should work as well. python driver.py | tee logs.txt. If not, then it’s probably an autoscaler bug.

1 Like