[BOHB] config variants generation

Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I am using Tune with BOHB.
My original search space contains both nested parameters (that is the original config object is a dictionary containing list of dicts among other key-value pairs), as well as conditional parameters.
I am encountering two issues at the moment:

  1. original tune distributions contained into a dict in a list are correctly sampled by the search generator. However, configs are not correctly merged back. Specifically,
ray/util/ml_utils/dict::deep_update

completely ignores the case of lists which are overwritten. As an example,

config = {
‘a’: 1,
‘b’: tune.choice([1, 2])
‘c’: [{
‘c1’: 1,
‘c2’: tune.uniform(0,1)
}]
}

results in a sampled configuration

config = {
‘a’: 1,
‘b’: 1 # randomly chosen
‘c’: [{
‘c2’: 0.1233 # randomly chosen, c1 has been deleted
}]
}

I saw that the merging functions are marked as deprecated. What is the plan with them?

  1. Original BOHB supports conditional parameters. Are you planning to include automatic conversion for such cases?

Cheers,

2

@tod88
Thanks for writing this up!
Could i get a bit more information on what you expect as the config variants?

So for

tune.run(train, config = {
	"a": 1,
	"b": tune.choice([1, 2]),
	"c": [{
	"c1": 1,
	"c2": tune.uniform(0,1)
}]}, num_samples=1)

I am seeing my trial taking in a config as

{'a': 1, 'b': 1, 'c': [{'c1': 1, 'c2': 0.008614192268075471}]}

which is what I would expect. Is this not what you would want?

3

@xwjiang2010
thanks for your answer. Apologies for the late reply.

Unfortunately this is not what I get.
Have you tries using BOHB as search algorithm?
As far as I understand looking inside the code, the config variants might be generated by the search_alg and my problem could arise from there.