How severe does this issue affect your experience of using Ray?
Medium: It contributes to significant difficulty to complete my task, but I can work around it
I am working with HyperOpt search algorithm and I have few doubts. HyperOpt requires knowledge of past results based on their respective hyperparameter for choosing the next hyperparameter combination. So , in ray how is done? and what does “n_initial_points” and “point_to_evaluate” states? I went through the documentation https://docs.ray.io/en/latest/tune/api_docs/suggestion.html#tune-hyperopt but still did not get clear picture.
For example, num_samples =20, and n_initial_points=4. Does hyperparamter chooses the initally the random hyperparameters for 4 different runs and based on that it chooses 5th run and soo on? or how does it work?
Hi @sai_lalith_Polawar,
generally yes, Hyperopt will sample n_initial_points randomly, and further trials will be informed by the results from previous trials.
However, the Ray Tune Hyperopt integration does not automatically limit concurrency - thus if you start 20 trials at the same time (because you maybe have the resources to do this on your cluster), the first n_initial_points will be randomly sampled and the others will be drawn from the TPE. But, since the TPE did not receive any results, yet, to fit the model, the estimates will be off (and probably also very similar).
Instead, you should use a ConcurrencyLimiter to limit the number of trials you’re running in parallel. Here you should probably specify that the number of concurrent trials should be lower or equal than n_initial_points.