# Ray Tune PBT - Structural Hyperparameters

**URL:** <https://discuss.ray.io/t/ray-tune-pbt-structural-hyperparameters/20525>\
**Category:** Ray Tune\
**Created:** [November 10, 2024, 12:08pm UTC](https://discuss.ray.io/t/ray-tune-pbt-structural-hyperparameters/20525 "2024-11-10T12:08:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jwickens](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jwickens/32/7290_2.png) [@jwickens](https://discuss.ray.io/u/jwickens)\
**Post date:** [November 10, 2024, 12:08pm UTC](https://discuss.ray.io/t/ray-tune-pbt-structural-hyperparameters/20525/1 "2024-11-10T12:08:33Z")

</div>

I’m using PBT to explore both typical hyperparameters and architectural ones:

```auto
config = {
    "lr": tune.uniform(1e-6, 1e-3),
    "weight_decay": tune.uniform(0.2, 0.01),
    "vision_layers": tune.choice([4, 8, 10]),
    "vision_width": tune.choice([32, 64, 128]),
    "vision_patch_size": tune.choice([8, 12]),
}
scheduler = PopulationBasedTraining(
    time_attr="training_iteration",
    metric="val_loss",
    mode="min",
    perturbation_interval=1,
    hyperparam_mutations={
        "lr": tune.loguniform(1e-5, 1e-3),
        "weight_decay": tune.loguniform(1e-6, 1e-4),
    },
)
tuner = tune.Tuner(
    trainable=TrainMetricsPBT,
    run_config=train.RunConfig(
        stop=stop_fn,
        checkpoint_config=train.CheckpointConfig(
            checkpoint_score_attribute="val_loss",
            checkpoint_score_order="min",
            num_to_keep=4,
        ),
    ),
    tune_config=tune.TuneConfig(
        reuse_actors=True,
        scheduler=scheduler,
        max_concurrent_trials=6,
        num_samples=20,
    ),
    param_space=config,
)

```

Since my concurrency is less than my sample size I expect multiplexing with PBT to explore other structural hyperparameter changes before the usual explore/explore with hyperparameter perbutations. Instead it only seems to sample the full param space up to my max\_concurrent\_trails of only 6.

Before I try something more complex like using a different scheduler for architecture search before using PBT for strict hyperparameter search I wanted to check that I wasnt’ missing something fundamental here.

Also if I may feedback that trials cloning other trials is super confusing. It would be much clearer if exploit simply terminated the old trial and started a new one with a new ID and everything, but copying the cloned trial checkpoint of course.

Thanks for the guidance.

---

<div class="post-metadata">

**Author:** ![jwickens](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jwickens/32/7290_2.png) [@jwickens](https://discuss.ray.io/u/jwickens)\
**Post date:** [November 15, 2024, 2:13pm UTC](https://discuss.ray.io/t/ray-tune-pbt-structural-hyperparameters/20525/2 "2024-11-15T14:13:06Z")

</div>

Nevermind… I upgraded to the latest Ray and PBT is now multiplexing correctly. Thank you!