Hi,
I was trying to use tune.run() with a nested config like this:
config = {
"seed": 1337
"epochs": 200
"batch_size": 16
"learning_rate": tune.grid_search([0.001, 0.003, 0.0001, 0.0003]),
"network_config": {
"num_filter_per_size": tune.grid_search([50, 150, 250, 350, 450, 550]),
"filter_sizes": tune.grid_search([[2], [4], [6], [8], [10]]),
"dropout": tune.grid_search([0.2, 0.4, 0.6, 0.8])
}
}
When passing in the nested config, parameters specified in the nested config are None in “Current best trial”. Please see the screenshot below.
The reason is that function best_trail_str is looking for config['network_config/dropout'] (i.e., p) that doesn’t exist. It should be unflattened to config['network_config']['dropout'].
def best_trial_str(
trial: Trial,
metric: str,
parameter_columns: Union[None, List[str], Dict[str, str]] = None):
"""Returns a readable message stating the current best trial."""
val = trial.last_result[metric]
config = trial.last_result.get("config", {})
parameter_columns = parameter_columns or list(config.keys())
if isinstance(parameter_columns, Mapping):
parameter_columns = parameter_columns.keys()
params = {p: config.get(p) for p in parameter_columns}
return f"Current best trial: {trial.trial_id} with {metric}={val} and " \
f"parameters={params}"
def _fair_filter_trials(trials_by_state: Dict[str, List[Trial]],
max_trials: int,
sort_by_metric: bool = False):
"""Filters trials such that each state is represented fairly.
The oldest trials are truncated if necessary.
Is passing in nested config an expected behavior for running tune.run()?
Thanks!
kai
December 13, 2021, 11:54am
2
For me it works as expected:
Current best trial: ffab5_00000 with _metric=5 and parameters={'seed': 1337, 'epochs': 200, 'batch_size': 16, 'learning_rate': 0.001, 'network_config': {'num_filter_per_size': 50}}
Which version of Ray are you using? Can you share more of your code with us or provide a reproducible example (running on the latest master or at least latest release)?
The fix should be easy (we’ll just use unflattened_lookup to get the right result), but to test this we need to be able to reproduce it.
1 Like
Hi @kai ,
Sure. The ray version is 1.9.0.
from ray import tune
def objective(step, alpha, beta):
return (0.1 + alpha * step / 100)**(-1) + beta * 0.1
def training_function(config):
alpha, beta = config["learning_rate"], config["network_config"]["num_filter_per_size"]
for step in range(10):
intermediate_score = objective(step, alpha, beta)
tune.report(mean_loss=intermediate_score)
metric, mode = 'mean_loss', 'min'
config = {
"seed": 1337,
"epochs": 200,
"batch_size": 16,
"learning_rate": tune.grid_search([0.001, 0.003, 0.0001, 0.0003]),
"network_config": {
"num_filter_per_size": tune.grid_search([50, 150, 250, 350, 450, 550])
}
}
# the `parameter_columns` is for simplifying the column names in the CLIReporter
# for example, `network_config/num_filter_per_size` -> `num_filter_per_size`
parameter_columns = {
'learning_rate': 'learning_rate',
'network_config/num_filter_per_size': 'num_filter_per_size'
}
reporter = tune.CLIReporter(metric_columns=['mean_loss'],
parameter_columns=parameter_columns,
metric=metric, mode=mode,
sort_by_metric=True)
analysis = tune.run(
training_function,
num_samples=1,
resources_per_trial={'cpu': 4, 'gpu': 1},
progress_reporter=reporter,
config=config)
print("Best config: ", analysis.get_best_config(metric=metric, mode=mode))
Thanks!!
kai
December 14, 2021, 4:57pm
4
Thanks for the repro script - this issue only comes up when using custom parameter_columns, hence why I didn’t see it first.
With the reproduction script I was able to fix this here: [tune] Fix best_trial_str for nested custom parameter columns by krfricke · Pull Request #21078 · ray-project/ray · GitHub
This fix should be included in the next Ray release (1.10.0). Thanks again for raising this!
1 Like
Great! Thanks again for the help!
