How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
I am using ASHA scheduler and BasicVariant Search Algo with the following early stopping configurations
scheduler = AsyncHyperBandScheduler(
metric=“composite_metric”,
mode=“min”,
max_t=max_num_epochs,
grace_period=2,
reduction_factor=2,
)
My resource R is epochs. My max_num_epochs is 10 and no of samples are also 10.
As I understand my 10 samples should go through 4 halving stages(rungs) specifically at 2,4,8,10 epochs. With the number of samples getting reduced in every stage by for eg 10,5,3,2.
This does not co-relate to the results I am getting.
looking at my results
Initial stage : 10 samples, all reach 2 epochs atleast because of grace period
1st halving stage: 5 samples are stopped at 2 epochs, 5 remaining are promoted to next stage (4 epochs)
2nd halving stage: 1 sample is stopped at 4 epochs, 4 are promoted to next stage (8 epochs)
3rd halving stage: 1 sample is stopped at 8 epochs, 3 are promoted to next final stage(10 epochs)
4th halving stage: all 3 sample run till 10 epochs, cannot proceed to more halving because max epochs reached.
Why are the samples not being successively halved according to the reduction factor of 2? I can see that the resource is doubling fine because of early stopping at 4 and 8 epochs but the no of sample being promoted seems not right except for the first stage of where 5 samples are stopped. Is it because of the grace period or am I missing something in my scheduler or my understanding of hyperband in general?