# GPU memory not cleared after trial

**URL:** <https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472>\
**Category:** Ray Tune\
**Created:** [September 7, 2021, 9:30pm UTC](https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472 "2021-09-07T21:30:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hahdawg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hahdawg/32/1439_2.png) [@hahdawg](https://discuss.ray.io/u/hahdawg)\
**Post date:** [September 7, 2021, 9:30pm UTC](https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472/1 "2021-09-07T21:30:41Z")

</div>

I’m doing hyperparameter optimization of a pytorch model using ray.tune, and I’m having an issue similar to the one described here:

[tensorflow - Out of memory at every second trial using Ray Tune - Stack Overflow](https://stackoverflow.com/questions/65722484/out-of-memory-at-every-second-trial-using-ray-tune)

I attempted to add the `wait_for_gpu` function, and according to the logs, gpu memory usage stays constant after 20 retries, at which time the function raises an error.

Is there a simple workaround here? Maybe something like the process described here:

[GPU Support — Ray v1.6.0](https://docs.ray.io/en/latest/using-ray-with-gpus.html)

in the section " Workers not Releasing GPU Resources", but for ray.tune?

Edit: Sleeping for 90s at the beginning of the objective function seems to have solved the issue, which makes me think there’s a problem with `wait_for_gpu`, because it was reporting constant gpu memory usage.

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [October 26, 2021, 8:19pm UTC](https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472/2 "2021-10-26T20:19:17Z")

</div>

Could you share a repro script? Curious to learn why trials are holding on to GPU resources.

---

<div class="post-metadata">

**Author:** ![hahdawg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/hahdawg/32/1439_2.png) [@hahdawg](https://discuss.ray.io/u/hahdawg)\
**Post date:** [January 13, 2022, 9:03pm UTC](https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472/3 "2022-01-13T21:03:50Z")

</div>

Apologies. Just saw this. Using a placement group factory fixed it.

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [January 18, 2022, 4:56pm UTC](https://discuss.ray.io/t/gpu-memory-not-cleared-after-trial/3472/4 "2022-01-18T16:56:05Z")

</div>

Nice! Thanks for reporting back.