What is the relationship between "trial scheduler" and "search algorithm"?

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Sorry if my question is stupid, but I am a newer to AutoML.
If search algorithm tells how to find the optimal superparameters why is trial scheduler neccessary, and if trial scheduler does why is search algorithm neccessary?
Or Ray Tune only works well when the according search algorithm and trial scheduler meets?

2

Hey @Ywandung-Lyou, allow me to explain.

Search algorithms (Searchers) suggest new configurations to try out. A new configuration is passed to a new Trial, which executes the training code (Trainable). Inside the Trainable, you report the obtained metrics. Depending on the type of a model, you may have access to metrics only at the end of the trial (eg. when you are tuning regularised regression), or you can access incremental metrics (eg. validation loss at every epoch when tuning a neural network). If you are reporting incremental results, then a scheduler can use those to determine whether a Trial should be early stopped or not. This is the ASHAScheduler, for example. The only exception is the default FIFOScheduler, which does not early stop trials (and therefore doesn’t require incremental results).

In short, a search algorithm suggests the configuration for the trial, and a scheduler determines whether a trial should be early stopped or not. Early stopping schedulers require you to report metrics to Tune multiple times during training. Searchers and schedulers serve different functions (though for example, BOHB is implemented as a scheduler and a searcher, and you have to use both to get the desired results).

1 Like
3

Hey Yard1, and thanks for your explanation.

In cifar10_pytorch, search space, scheduler and num_samples are defined but search algorithm is not. It means that search algorithm and scheduler can be used solo, right?

Now I have invented a new text-matching deep-learning model structure and I want to find the optimal parameters. I have defined the according search space, but I do not know how to use ASHA. Can I just use the ASHAScheduler defined in cifar10_pytorch, or I have to read the paper and define my own scheduler?

4

If a search algorithm is not defined, the default BasicVariantGenerator searcher is used, which is simple random/grid search. Correspondingly, if the scheduler is not defined, the default, non-early stopping FIFOScheduler is used.

In order to use ASHA or any other early stopping scheduler, you need to have a Trainable that reports metrics multiple times (eg. in a loop, once per epoch), and set the following:

  • metric (name) parameter in tune.run (or the scheduler itself, but tune.run is recommended)
  • mode parameter in tune.run (or the scheduler itself, but tune.run is recommended)
  • for ASHA specifically, max_t parameter in the scheduler, corresponding to the maximum number of iterations in the trainable (eg. max epochs)

You can use the built-in ASHA scheduler just fine, just as in the cifar example.

1 Like