[Tune] How are schedulers interfering with stoppers

Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around

Hi Ray community,

I implemented some custom stopper MyStopper inheriting from ray.tune.stopper.Stopper and would like to combine that with the “classic” stopping criterion for maximum iteration count.

What I observe is that ASHAScheduler overrules the criteria defined in the stoppers, while simple FIFOScheduler has the tendency to ignore the MyStopper() and take only the maximum iteration one.

Now here my questions:

  1. Which kind of hierarchy is applied between schedulers and stoppers?
  2. What is the best practise where to define which kind of experiment control?

Thank you for your advice!

scheduler = ASHAScheduler(
    metric="episode_reward_mean",  # The metric to optimize
    mode="max",  # Maximize the reward
    max_t=100,  # Max number of training iterations
    grace_period=10,  # Minimum number of iterations before stopping
    reduction_factor=2,  # Scaling factor for trials
)

tuner = tune.Tuner(
    "PPO",
    param_space=config,
    run_config=air.RunConfig(
        name="curr-experiment",  
        stop=CombinedStopper(
            MyStopper(), MaximumIterationStopper(100)
        ),
    ),
    tune_config=tune.TuneConfig(
        num_samples=2,
        max_concurrent_trials=2,
        scheduler=scheduler,
    ),
)