Stop programmatically using running time

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Is it possible to stop the run if a tune has been running more than a certain amount of time? More specifically, if there’s a way for ray to access the running time from, e.g. slum jobs, local interactive runs, so that one can pass it to a stopper?

2

There are two possibilities here:

  1. Set time_budget_s within TuneConfig(time_budget_s=max_seconds). This is a global time budget in seconds that the Tune experiment will run for. After this many seconds, all trials will be stopped.

  2. time_total_s is reported as a default metric by Tune, which is the number of seconds elapsed since the start of the experiment. You can try something like: stop={"time_total_s": max_seconds}. However, this timer is on a per-trial basis, so option 1 may be what you want in this case.

This thread asks a similar question: How to use time_total_s as a stop condition?.

1 Like