Hello,
I understand that this question will be vague, and it is mainly though to my ignorance on the PBT mechanism.
I am trying to tune a CNN model on genomic sequence data on a relatively easy task: the first model I came up with in 5 minutes achieved a 0.92 AUPRC and 0.94 AUROC validation scores
After having established that the task is relatively easy (hence most models will achieve a decent result), I wanted to try the PBT method to tune the meta-model to learn how to use PBT.
Having defined the parameters as the number of filters/kernel size plus some activation regularization weight, I have tried to use the PBT as follows:
First I have defined a space of parameters using uniform lambdas as shown in the tutorial here. Here I use double lambdas just to capture the local variables defining the range.
space = {
key: (lambda vr: lambda: np.random.uniform(*vr))(val_range)
for key, val_range in model.space().items()
}
Then I define the train method as follows:
def train_convnet(config):
import silence_tensorflow.auto
window_size = 256
train, test = create_training_sequence(window_size)
meta_model: Model = build_model(window_size)
meta_model.space()
model = meta_model.build(**config)
model.compile(
optimizer='nadam',
loss="binary_crossentropy",
metrics=[
"accuracy",
AUC(curve="PR", name="auprc"),
AUC(curve="ROC", name="auroc")
]
)
model.fit(
train,
validation_data=test,
epochs=1000,
verbose=False,
callbacks=[
TuneReportCallback(metrics="val_auprc"),
EarlyStopping(monitor="auprc", patience=5)
]
)
And finally I call the tuning process:
from ray.tune.stopper import EarlyStopping as TuneEarlyStopping
scheduler = PopulationBasedTraining(
time_attr="training_iteration",
perturbation_interval=5,
hyperparam_mutations=space
)
analysis = tune.run(
train_convnet,
name="pbt_test",
scheduler=scheduler,
metric="val_auprc",
mode="max",
verbose=1,
stop=TuneEarlyStopping("val_auprc", ),
resources_per_trial={
"cpu": cpu_count()//4,
"gpu": 1
},
num_samples=500
)
The tuning processes are then all terminated and they achieve at most an AUPRC of 0.45.
What am I doing wrong? What information is needed to properly resolve this issue?
Though to the reserved nature of the training labels, I cannot share an example of the dataset but I believe that the issue at hand has little to do with the considered task and more to do with how I am using tune and PBT.
Thank you,
Luca
