Hi @pchalasani, I think there are a few things to clarify here.
First, I would suggest to use tune.grid_search([0, 1]) instead of tune.choice([0, 1]). With choice you get a random seleciton - thus all trial could be a=0! (I had this when running your script). If you do this, set num_samples=2 to have 4 trials to run (2 times the full grid search).
Second, early termination with ASHA always uses the latest available result. This means if your trials report 10 and 3 after the 5th step (when they do the first evaluation), ASHA will stop the trials with a=1 (which reported 3) early.
Third, in experiment analysis the property best_result means “take the trial with the best last score and return the result”. Thus, this is a very simple lookup - and it always returns the last reported score.
This also means that in the example you get a=1 and max_score=3 because the trials that continue to run (a=0) eventually report a score of 1, which is lower than 3. Tune does not take the number of iterations into account here, or the fact that the trial was early stopped.
However, you can use other methods of experiment analysis to make different analyses. For instance, you can specify the scope in get_best_trial() to get the best ever observed result (instead of the last):
best_trial_in_run = analysis.get_best_trial("score", "max", scope="all")
print(f"Best result ever was achieved by trial {best_trial_in_run}")
And you can also get a dataframe with the best observed results for further analysis:
df = analysis.dataframe("score", "max")
print(df)
WIth the results:
best score was 3 with a = 1
Best result ever was achieved by trial trainable_aefb5_00000
score ... logdir
0 10.0 ... /Users/kai/ray_results/trainable_2022-03-25_13...
1 4.0 ... /Users/kai/ray_results/trainable_2022-03-25_13...
2 10.0 ... /Users/kai/ray_results/trainable_2022-03-25_13...
3 4.0 ... /Users/kai/ray_results/trainable_2022-03-25_13...