# Terminated Trials hold memory causing OOM

**URL:** <https://discuss.ray.io/t/terminated-trials-hold-memory-causing-oom/3133>\
**Category:** Ray Tune\
**Created:** [August 5, 2021, 7:28am UTC](https://discuss.ray.io/t/terminated-trials-hold-memory-causing-oom/3133 "2021-08-05T07:28:35Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nijkah](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nijkah/32/1327_2.png) [@nijkah](https://discuss.ray.io/u/nijkah)\
**Post date:** [August 5, 2021, 7:28am UTC](https://discuss.ray.io/t/terminated-trials-hold-memory-causing-oom/3133/1 "2021-08-05T07:28:35Z")

</div>

![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/e/eed9254606f62467f94e6dea248c9cf94d4bd9b8.png)

Until 24/64 trials, tuner is normally executed.  
But if it exceeds a certain number of trials, all trials raise error for RayOutOfMemoryError.

I use Trainable class, so I am using cleanup function to delete variables explicitly like below

```auto
def cleanup(self):
    del self.runner
    del self.model
    del self.optimizer
    del self.train_data_loaders
    del self.valid_dataset
    del self.valid_iter
    del self.config
    import gc; gc.collect()
    torch.cuda.empty_cache()

```

And the number of simultaneous trials can be up to 4 since there are 4 gpus, and each process requires 1 gpu.

I would thank if you give any advices to troubleshoot.

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [October 26, 2021, 8:38pm UTC](https://discuss.ray.io/t/terminated-trials-hold-memory-causing-oom/3133/2 "2021-10-26T20:38:37Z")

</div>

Hi,  
Could you provide a repro script that I can help take a look at?