How to tune hyperparameters such as layer count/sizes of inner layers/dropout probabilities with Huggingface Transformers

The Huggingface Documentation makes mention that I can tune parameters like layer count, sizes of inner layers, dropout probabilities of a transformers model with a Ray Tune trial object by passing it to the user-defined model_init function.

How exactly would the implementation look for this? Perhaps using this example as a reference.

Hey @Luca_Guarro , you can make model_init take a config parameter. For example, in the script above, you can do:

    def get_model(params):
        return AutoModelForSequenceClassification.from_pretrained(
            model_name,
            config=config + params,
        )

...
Trainer(model_init=get_model, ...)