# Error related to 'performance bottleneck' and 'start\_trial'

**URL:** <https://discuss.ray.io/t/error-related-to-performance-bottleneck-and-start-trial/3671>\
**Category:** Uncategorized\
**Created:** [September 29, 2021, 3:15pm UTC](https://discuss.ray.io/t/error-related-to-performance-bottleneck-and-start-trial/3671 "2021-09-29T15:15:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![BttMA](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bttma/32/1559_2.png) [@BttMA](https://discuss.ray.io/u/BttMA)\
**Post date:** [September 29, 2021, 3:15pm UTC](https://discuss.ray.io/t/error-related-to-performance-bottleneck-and-start-trial/3671/1 "2021-09-29T15:15:51Z")

</div>

Hello,  
It is my first time that I use Ray and I like it how it is helping in fine tuning a very complex models.  
I am using Ray to my fine tune my model (Bert) but I keep getting the bottleneck WARNING (`2021-09-29 14:51:42,804	WARNING util.py:164 -- The `start\_trial` operation took 2.259 s, which may be a performance bottleneck.`). I tried every possible solution I found here or on you website (btw, the FAQ section 22 (How can I avoid bottlenecks?) is very complicated to follow, It would be helpful if you add some exemple on the proposed solutions) but with no help!

And It does not stop there. In the `resources_per_trial` argument, when I pass `get_tune_ddp_resources(num_workers=4)` (This one I copy paste it just for testing from you gitHub [here](https://github.com/ray-project/ray_lightning/blob/main/README.md#hyperparameter-tuning-with-ray-tune) )

```auto
analysis = tune.run(
        model,
        metric="loss",
        mode="min",
        config=config,
        num_samples=10,
        resources_per_trial=get_tune_ddp_resources(num_workers=4),
        name="tune_mnist")

```

I got this results and it remains “PENDING” for hours with no “RUNUNG” or anything!

| Trial name | status | loc | batch\_size | lr |
| --- | --- | --- | --- | --- |
| model\_0195b\_00000 | PENDING | | 8 | 0.0121142 |
| model\_0195b\_00001 | PENDING | | 4 | 0.000707256 |
| model\_0195b\_00002 | PENDING | | 4 | 0.000810177 |
| model\_0195b\_00003 | PENDING | | 4 | 0.0134884 |
| model\_0195b\_00004 | PENDING | | 4 | 0.000635871 |
| model\_0195b\_00005 | PENDING | | 8 | 0.000206381 |
| model\_0195b\_00006 | PENDING | | 8 | 0.0173723 |
| model\_0195b\_00007 | PENDING | | 4 | 0.000157989 |
| model\_0195b\_00008 | PENDING | | 8 | 0.000697227 |
| model\_0195b\_00009 | PENDING | | 8 | 0.0356216 |

And when I change it to gpu (`resources_per_trial={'gpu': 1}`), I get an error in the last stat and also it remains “PENDING” for hours.

| Trial name | status | loc | batch\_size | lr |
| --- | --- | --- | --- | --- |
| model\_278c7\_00001 | PENDING | | 4 | 0.0186282 |
| model\_278c7\_00002 | PENDING | | 4 | 0.037845 |
| model\_278c7\_00003 | PENDING | | 8 | 0.00038114 |
| model\_278c7\_00004 | PENDING | | 4 | 0.000118839 |
| model\_278c7\_00005 | PENDING | | 4 | 0.000281904 |
| model\_278c7\_00006 | PENDING | | 8 | 0.00388556 |
| model\_278c7\_00007 | PENDING | | 8 | 0.00101594 |
| model\_278c7\_00008 | PENDING | | 8 | 0.000134712 |
| model\_278c7\_00009 | PENDING | | 8 | 0.00624219 |
| model\_278c7\_00000 | **ERROR** | | 8 | 0.000103422 |

Below is a snippet of my code. I hope it could help you find the error and where it cames from.

```auto
from ray.tune.integration.pytorch_lightning import TuneReportCallback
callback = TuneReportCallback({
    "avg_val_loss": "avg_val_loss", 
    "avg_accuracy": "avg_accuracy"},
    on="validation_end") # I am using pytorch lightning so these are the results of my model from the validation_epoch_end method

config = {
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([4, 8]),
}

def train_tune(config, epochs=N_EPOCHS, gpus=1):
  model = CommentClassifier(config, n_warmup_steps=warmup_steps, n_training_steps=total_training_steps)

  trainer = pl.Trainer(
    max_epochs=epochs,
    gpus=gpus,
    progress_bar_refresh_rate=0,
    callbacks=[callback],
    logger=logger)
  
  trainer.fit(model, dataloader_train, dataloader_validation)

analysis = tune.run(
        model,
        metric="loss",
        mode="min",
        config=config,
        num_samples=10,
        resources_per_trial={'gpu': 1}, #get_tune_ddp_resources(num_workers=4)
        name="tune_mnist")

print("Best hyperparameters: ", analysis.best_config)

```