[Tune FAQ] Formula for estimating iteration & timesteps count

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

After being overwhelmed of the fine details between ray.train() and ray.tune(), I am now digging deeper in the documentation and taking time for step-by-step-examples to understand and learn.

Nevertheless, I think it would be very helpful for the official Tune FAQ to provide a formula approach how the minimum size of tune trials is dependent on the configs for training / tuning / trials.

As a starting point, I would like to following theoretically determined formula based on the documentation:
num_samples * matrix product of tune search space (e.g. 3*3=9) * train_batch_size

My currently running PPO trial without any tune search spaces proves me that this formula is non-exhaustive, or rather a first idea.

For sure, users can interfere with stopping criteria. But what if stopping criteria are not defined? Does is run endless in that case?

Happy to hear your feedback and share ideas on how it could like!

2

Hi @PhilippWillms,

sorry to hear you felt overwhelmed! We appreciate any feedback or suggestions that will make it easier for users to onboard.

Generally the number of trials is just num_samples. This only changes if you use grid search, as documented in the API reference.

In that case, it will be total_num_samples = num_samples * [matrix product of grid search parameters].

You can also set time_budget_s and num_samples=-1 to run an infinite amount of samples (until the time runs out). If you don’t set a time_budget_1, it will run until you cancel the script.

I’m happy to add a section in the FAQ for this. Is there any other information you’d like to have included?