How to tune hyperparameters such as layer count/sizes of inner layers/dropout probabilities with Huggingface Transformers

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

The Huggingface Documentation makes mention that I can tune parameters like layer count, sizes of inner layers, dropout probabilities of a transformers model with a Ray Tune trial object by passing it to the user-defined model_init function.

How exactly would the implementation look for this? Perhaps using this example as a reference.

2

Hey @Luca_Guarro , you can make model_init take a config parameter. For example, in the script above, you can do:

    def get_model(params):
        return AutoModelForSequenceClassification.from_pretrained(
            model_name,
            config=config + params,
        )

...
Trainer(model_init=get_model, ...)
3

Do I need to do anything in particular to make sure the parameters actually get passed?

I defined the function like so:

def get_model(params):
        db_config = db_config_base
        db_config.update({'alpha': params['alpha_val'], 'dropout': params['dropout_val']})
        return DistilBERTForMultipleSequenceClassification.from_pretrained(db_config, num_labels1 = 2, num_labels2 = 8)

But then when I run it, I get the error:

TypeError: ‘NoneType’ object is not subscriptable

Aka ‘Params’ is none. So how do I actually ensure that the get_model function receives an instance of the ray tune trial object?

4

Can you provide a full script that you’re running? I’d love to try to take a look at this myself on my computer.

5

Hi Richard, we actually solved this issue on this forum. Let me know if that suffices