TrialPlateauStopper improvement

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hello,

I suggest to add to this stopper, on top of metric_threshold and mode (which stop per trial) also a global metric_threshold_exp and mode_exp (which will stop the whole experiment).

Sometimes it’s good to stop the whole tune process if a global minima (or close to that) was found.
Something similar with this:

class BestValueStopper(Stopper):
    def __init__(self, target_value: int = 0, metric="", mode="", min_iterations=10):
        self._target_value = target_value
        self._metric = metric
        self._mode = mode
        self._min_iterations = min_iterations
        self._results = []
        self._should_stop = False
        if mode == "min":
            self._mode_op = np.less_equal
        else:
            self._mode_op = np.greater_equal

    def __call__(self, trial_id, result):
        self._results.append(trial_id)
        if  self._mode_op(result[self._metric], self._target_value) and len(self._results) >= self._min_iterations:  #  
            self._should_stop = True
            logger.info(
                f"BestValueStopper should stop - iter {len(self._results)} - {result[self._metric]} / {self._target_value}"
            )
            return True
        return self._should_stop

    def stop_all(self):
        return self._should_stop
2

Hm, could you just use an (additional) ExperimentPlateauStopper for this?

3

Yes, you are right. Not sure why I did not choose that one. Thanks!

4

But anyhow, I believe also an absolute value will be good there, aside std dev

5

Hello,
I changed a bit the ExperimentPlateauStopper because in some cases the std does not meet the criteria, even when self._top_values does not change after patience iterations.

diff --git a/python/ray/tune/stopper.py b/python/ray/tune/stopper.py
index bc0940bd7…bbd849e35 100644
— a/python/ray/tune/stopper.py
+++ b/python/ray/tune/stopper.py
@@ -183,6 +183,7 @@ class ExperimentPlateauStopper(Stopper):
self._std = std
self._top = top
self._top_values = []

  •    self._old_mean = 0

    def call(self, trial_id, result):
    “”“Return a boolean representing if the tuning has to stop.”""
    @@ -205,8 +206,11 @@ class ExperimentPlateauStopper(Stopper):
    return self.stop_all()

    def has_plateaued(self):

  •    return (len(self._top_values) == self._top

  •            and np.std(self._top_values) <= self._std)

  •    result = (len(self._top_values) == self._top) and (

  •        np.std(self._top_values) <= self._std or np.mean(self._top_values) == self._old_mean

  •    )

  •    self._old_mean = np.mean(self._top_values)

  •    return result

    def stop_all(self):
    “”“Return whether to stop and prevent trials from starting.”""