Best trail printing None when input a nested config

Hi,

I was trying to use tune.run() with a nested config like this:

config = {
    "seed": 1337
    "epochs": 200
    "batch_size": 16
    "learning_rate": tune.grid_search([0.001, 0.003, 0.0001, 0.0003]),
    "network_config": {
         "num_filter_per_size": tune.grid_search([50, 150, 250, 350, 450, 550]),
         "filter_sizes": tune.grid_search([[2], [4], [6], [8], [10]]),
         "dropout":  tune.grid_search([0.2, 0.4, 0.6, 0.8])
    }
}

When passing in the nested config, parameters specified in the nested config are None in “Current best trial”. Please see the screenshot below.

The reason is that function best_trail_str is looking for config['network_config/dropout'] (i.e., p) that doesn’t exist. It should be unflattened to config['network_config']['dropout'].

Is passing in nested config an expected behavior for running tune.run()?
Is there any suggestion for using the nested config?

Thanks!

For me it works as expected:

Current best trial: ffab5_00000 with _metric=5 and parameters={'seed': 1337, 'epochs': 200, 'batch_size': 16, 'learning_rate': 0.001, 'network_config': {'num_filter_per_size': 50}}

Which version of Ray are you using? Can you share more of your code with us or provide a reproducible example (running on the latest master or at least latest release)?

The fix should be easy (we’ll just use unflattened_lookup to get the right result), but to test this we need to be able to reproduce it.

1 Like

Hi @kai,

Sure. The ray version is 1.9.0.
Please see the example below:

Code

from ray import tune


def objective(step, alpha, beta):
    return (0.1 + alpha * step / 100)**(-1) + beta * 0.1


def training_function(config):
    alpha, beta = config["learning_rate"], config["network_config"]["num_filter_per_size"]
    for step in range(10):
        intermediate_score = objective(step, alpha, beta)
        tune.report(mean_loss=intermediate_score)


metric, mode = 'mean_loss', 'min'
config = {
    "seed": 1337,
    "epochs": 200,
    "batch_size": 16,
    "learning_rate": tune.grid_search([0.001, 0.003, 0.0001, 0.0003]),
    "network_config": {
        "num_filter_per_size": tune.grid_search([50, 150, 250, 350, 450, 550])
    }
}
# the `parameter_columns` is for simplifying the column names in the CLIReporter
# for example, `network_config/num_filter_per_size` -> `num_filter_per_size` 
parameter_columns = {
      'learning_rate': 'learning_rate', 
      'network_config/num_filter_per_size': 'num_filter_per_size'
}
reporter = tune.CLIReporter(metric_columns=['mean_loss'],
                            parameter_columns=parameter_columns,
                            metric=metric, mode=mode,
                            sort_by_metric=True)
analysis = tune.run(
        training_function,
        num_samples=1,
        resources_per_trial={'cpu': 4, 'gpu': 1},
        progress_reporter=reporter,
        config=config)
print("Best config: ", analysis.get_best_config(metric=metric, mode=mode))

Screenshot

Thanks!!

Thanks for the repro script - this issue only comes up when using custom parameter_columns, hence why I didn’t see it first.

With the reproduction script I was able to fix this here: [tune] Fix best_trial_str for nested custom parameter columns by krfricke · Pull Request #21078 · ray-project/ray · GitHub

This fix should be included in the next Ray release (1.10.0). Thanks again for raising this!

1 Like

Great! Thanks again for the help! :grin: