# GPU Memory not clearing after one Ray tune task

**URL:** <https://discuss.ray.io/t/gpu-memory-not-clearing-after-one-ray-tune-task/12134>\
**Category:** Uncategorized\
**Created:** [September 14, 2023, 4:39pm UTC](https://discuss.ray.io/t/gpu-memory-not-clearing-after-one-ray-tune-task/12134 "2023-09-14T16:39:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ksarullo](https://avatars.discourse-cdn.com/v4/letter/k/4bbf92/32.png) [@ksarullo](https://discuss.ray.io/u/ksarullo)\
**Post date:** [September 14, 2023, 4:39pm UTC](https://discuss.ray.io/t/gpu-memory-not-clearing-after-one-ray-tune-task/12134/1 "2023-09-14T16:39:25Z")

</div>

I’m doing hyperparameter optimization of a tensorflow model using ray.tune, a similar task to the one posted here:

> [@GPU memory not cleared after trial](https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472):
>
> I’m doing hyperparameter optimization of a pytorch model using ray.tune, and I’m having an issue similar to the one described here: [tensorflow - Out of memory at every second trial using Ray Tune - Stack Overflow](https://stackoverflow.com/questions/65722484/out-of-memory-at-every-second-trial-using-ray-tune) I attempted to add the wait\_for\_gpu function, and according to the logs, gpu memory usage stays constant after 20 retries, at which time the function raises an error. Is there a simple workaround here? Maybe something like the process described here: [GPU Support — Ray v1.6.0](https://docs.ray.io/en/latest/using-ray-with-gpus.html) in the…

I have resources={‘cpu’: 1, ‘gpu’:1} however the gpu memory is not being cleared after running one of the tasks. I have tried sleeping and wait\_for\_gpu but the memory never clears. Any tips on how to fix this?

---

<div class="post-metadata">

**Author:** ![hahdawg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hahdawg/32/1439_2.png) [@hahdawg](https://discuss.ray.io/u/hahdawg)\
**Post date:** [September 14, 2023, 7:14pm UTC](https://discuss.ray.io/t/gpu-memory-not-clearing-after-one-ray-tune-task/12134/2 "2023-09-14T19:14:04Z")

</div>

Try doing

```python
from ray.tune.execution.placement_groups import PlacementGroupFactory
resources=PlacementGroupFactory([{"CPU": 1, "GPU": 1}])

```

Then pass the resources variable to `tune.run` instead of a dict.

Also, at the end of my objective function, I used `torch.cuda.empty_cache()` to clear GPU memory. I haven’t used tensorflow for a long time, but there must be an option to do that.

---

<div class="post-metadata">

**Author:** ![ksarullo](https://avatars.discourse-cdn.com/v4/letter/k/4bbf92/32.png) [@ksarullo](https://discuss.ray.io/u/ksarullo)\
**Post date:** [September 14, 2023, 8:33pm UTC](https://discuss.ray.io/t/gpu-memory-not-clearing-after-one-ray-tune-task/12134/3 "2023-09-14T20:33:19Z")

</div>

Sadly that didn’t work. I’ve also tried the only alternative for torch.cuda.empty\_cache() in tensorflow which is to close the session and reset the graph I believe which didn’t work either.