# Could not find best trial. Did you pass the correct \`metric\` parameter?

**URL:** <https://discuss.ray.io/t/could-not-find-best-trial-did-you-pass-the-correct-metric-parameter/4440>\
**Category:** Ray Tune\
**Created:** [December 14, 2021, 12:53pm UTC](https://discuss.ray.io/t/could-not-find-best-trial-did-you-pass-the-correct-metric-parameter/4440 "2021-12-14T12:53:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rienboonstoppel](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rienboonstoppel/32/1893_2.png) [@rienboonstoppel](https://discuss.ray.io/u/rienboonstoppel)\
**Post date:** [December 14, 2021, 12:53pm UTC](https://discuss.ray.io/t/could-not-find-best-trial-did-you-pass-the-correct-metric-parameter/4440/1 "2021-12-14T12:53:16Z")

</div>

Hi,

I am new to Ray Tune, but I cannot figure out my mistake.  
I am using Pytorch Lightning with the TuneReportCallback in order to pass the loss for Tune. However, it is not having it… Whatever I try I cannot get it to work. It runs perfectly fine, but at the end everytime the following is displayed:  
`Could not find best trial. Did you pass the correct `metric` parameter?`

I stumbled upon this topic: [Could not find best trial](https://discuss.ray.io/t/could-not-find-best-trial/295)  
And i tried to replicate that by adding manually in the training loop a `tune.report(...)`, but that didn’t do anything as well. What am I doing wrong?

EDIT: I did some more digging, and it seems that ‘progress.csv’ stays empty, maybe that has something to do with it? And in Tensorboard there is no ‘ray’ sub-header, whereas in the Lightning-MNIST example there is…

The relevant code (I think) I use is as follows:  
Trainer:

```auto
class LitTrainer(pl.LightningModule):
(...)
    def validation_epoch_end(self, outputs):
        avg_loss = torch.mean(torch.stack(outputs))
        self.log("val_loss", avg_loss, sync_dist=True)
        # tune.report(loss=avg_loss)

```

In main file:

```auto
def train_tune(config, args):
(...)
    logger = TensorBoardLogger(save_dir=tune.get_trial_dir(), name="", version=".")
    model = LitTrainer(netG=generator, config=config, args=args)

    trainer = pl.Trainer(
        gpus=args.gpus,
        max_epochs=args.max_epochs,
        logger=logger,
        log_every_n_steps=100,
        strategy='ddp_spawn',
        precision=args.precision,
        callbacks=[
            TuneReportCallback(
                {
                    "loss": "val_loss",
                },
                on="validation_end"),
        ],
        enable_progress_bar=False,
    )

    trainer.fit(
        model,
    )

def main():
    config = {
        'learning_rate': tune.loguniform(1e-4, 1e-2),
    }

    scheduler = ASHAScheduler(
        max_t=args.max_epochs,
        grace_period=1,
        reduction_factor=2)

    reporter = CLIReporter(
        parameter_columns=["learning_rate"],
        metric_columns=["loss", "training_iteration"])

    resources_per_trial = {'cpu': 24, 'gpu': args.gpus}

    train_fn_with_parameters = tune.with_parameters(
        train_tune,
        args=args
        )

    analysis = tune.run(
        train_fn_with_parameters,
        resources_per_trial=resources_per_trial,
        metric="loss",
        mode="min",
        config=config,
        num_samples=args.num_samples,
        scheduler=scheduler,
        progress_reporter=reporter,
        name='tune_test',
    )

    print('Best hyperparameters found were: ', analysis.best_config)

```

So what am I doing wrong?

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [December 17, 2021, 1:48am UTC](https://discuss.ray.io/t/could-not-find-best-trial-did-you-pass-the-correct-metric-parameter/4440/2 "2021-12-17T01:48:15Z")

</div>

Hey @rienboonstoppel ,

Based on the topic you linked, the issue could be that you’re reporting a Tensor:

```auto
avg_loss = torch.mean(torch.stack(outputs))

```

Could you try the suggested solution of converting this to a float?

---

<div class="post-metadata">

**Author:** ![rienboonstoppel](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rienboonstoppel/32/1893_2.png) [@rienboonstoppel](https://discuss.ray.io/u/rienboonstoppel)\
**Post date:** [December 17, 2021, 9:26am UTC](https://discuss.ray.io/t/could-not-find-best-trial-did-you-pass-the-correct-metric-parameter/4440/3 "2021-12-17T09:26:48Z")

</div>

Hi @matthewdeng,  
No, it is not that it has problems with a Tensor. I tried the things that where mentioned in the linked topic, but that did not solve anything.  
I figured it out just yet. Apparently Ray Tune is not able to cope with multipe GPUs. Because when I increase the number of GPUs it is not able to get the metrics. As strategy I tried both ddp and ddp\_spawn, but that did not make a difference. So that is a bit of a bummer. But instead of training one model on multipe GPUs at a time, I will train multiple models on a single GPU parallel. Total runtime should be quite similar I think.

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [December 17, 2021, 5:10pm UTC](https://discuss.ray.io/t/could-not-find-best-trial-did-you-pass-the-correct-metric-parameter/4440/4 "2021-12-17T17:10:11Z")

</div>

Ah I see, can you try using the PyTorch Lightning [RayPlugin](https://docs.ray.io/en/master/ray-lightning.html#pytorch-distributed-data-parallel-plugin-on-ray)?