Hi,
I was trying to use tune.run()
with a nested config like this:
config = {
"seed": 1337
"epochs": 200
"batch_size": 16
"learning_rate": tune.grid_search([0.001, 0.003, 0.0001, 0.0003]),
"network_config": {
"num_filter_per_size": tune.grid_search([50, 150, 250, 350, 450, 550]),
"filter_sizes": tune.grid_search([[2], [4], [6], [8], [10]]),
"dropout": tune.grid_search([0.2, 0.4, 0.6, 0.8])
}
}
When passing in the nested config, parameters specified in the nested config are None
in “Current best trial”. Please see the screenshot below.
The reason is that function best_trail_str
is looking for config['network_config/dropout']
(i.e., p
) that doesn’t exist. It should be unflattened to config['network_config']['dropout']
.
def best_trial_str(
trial: Trial,
metric: str,
parameter_columns: Union[None, List[str], Dict[str, str]] = None):
"""Returns a readable message stating the current best trial."""
val = trial.last_result[metric]
config = trial.last_result.get("config", {})
parameter_columns = parameter_columns or list(config.keys())
if isinstance(parameter_columns, Mapping):
parameter_columns = parameter_columns.keys()
params = {p: config.get(p) for p in parameter_columns}
return f"Current best trial: {trial.trial_id} with {metric}={val} and " \
f"parameters={params}"
def _fair_filter_trials(trials_by_state: Dict[str, List[Trial]],
max_trials: int,
sort_by_metric: bool = False):
"""Filters trials such that each state is represented fairly.
The oldest trials are truncated if necessary.
Is passing in nested config an expected behavior for running tune.run()
?
Is there any suggestion for using the nested config?
Thanks!
kai
December 13, 2021, 11:54am
2
For me it works as expected:
Current best trial: ffab5_00000 with _metric=5 and parameters={'seed': 1337, 'epochs': 200, 'batch_size': 16, 'learning_rate': 0.001, 'network_config': {'num_filter_per_size': 50}}
Which version of Ray are you using? Can you share more of your code with us or provide a reproducible example (running on the latest master or at least latest release)?
The fix should be easy (we’ll just use unflattened_lookup
to get the right result), but to test this we need to be able to reproduce it.
1 Like
Hi @kai ,
Sure. The ray version is 1.9.0.
Please see the example below:
Code
from ray import tune
def objective(step, alpha, beta):
return (0.1 + alpha * step / 100)**(-1) + beta * 0.1
def training_function(config):
alpha, beta = config["learning_rate"], config["network_config"]["num_filter_per_size"]
for step in range(10):
intermediate_score = objective(step, alpha, beta)
tune.report(mean_loss=intermediate_score)
metric, mode = 'mean_loss', 'min'
config = {
"seed": 1337,
"epochs": 200,
"batch_size": 16,
"learning_rate": tune.grid_search([0.001, 0.003, 0.0001, 0.0003]),
"network_config": {
"num_filter_per_size": tune.grid_search([50, 150, 250, 350, 450, 550])
}
}
# the `parameter_columns` is for simplifying the column names in the CLIReporter
# for example, `network_config/num_filter_per_size` -> `num_filter_per_size`
parameter_columns = {
'learning_rate': 'learning_rate',
'network_config/num_filter_per_size': 'num_filter_per_size'
}
reporter = tune.CLIReporter(metric_columns=['mean_loss'],
parameter_columns=parameter_columns,
metric=metric, mode=mode,
sort_by_metric=True)
analysis = tune.run(
training_function,
num_samples=1,
resources_per_trial={'cpu': 4, 'gpu': 1},
progress_reporter=reporter,
config=config)
print("Best config: ", analysis.get_best_config(metric=metric, mode=mode))
Screenshot
Thanks!!
kai
December 14, 2021, 4:57pm
4
Thanks for the repro script - this issue only comes up when using custom parameter_columns
, hence why I didn’t see it first.
With the reproduction script I was able to fix this here: [tune] Fix best_trial_str for nested custom parameter columns by krfricke · Pull Request #21078 · ray-project/ray · GitHub
This fix should be included in the next Ray release (1.10.0). Thanks again for raising this!
1 Like
Great! Thanks again for the help!