GPU Memory not clearing after one Ray tune task

[ksarullo]

I’m doing hyperparameter optimization of a tensorflow model using ray.tune, a similar task to the one posted here:

GPU memory not cleared after trial

I’m doing hyperparameter optimization of a pytorch model using ray.tune, and I’m having an issue similar to the one described here: tensorflow - Out of memory at every second trial using Ray Tune - Stack Overflow I attempted to add the wait_for_gpu function, and according to the logs, gpu memory usage stays constant after 20 retries, at which time the function raises an error. Is there a simple workaround here? Maybe something like the process described here: GPU Support — Ray v1.6.0 in the…

I have resources={‘cpu’: 1, ‘gpu’:1} however the gpu memory is not being cleared after running one of the tasks. I have tried sleeping and wait_for_gpu but the memory never clears. Any tips on how to fix this?

[hahdawg]

Try doing

from ray.tune.execution.placement_groups import PlacementGroupFactory
resources=PlacementGroupFactory([{"CPU": 1, "GPU": 1}])

Then pass the resources variable to tune.run instead of a dict.

Also, at the end of my objective function, I used torch.cuda.empty_cache() to clear GPU memory. I haven’t used tensorflow for a long time, but there must be an option to do that.

[ksarullo]

Sadly that didn’t work. I’ve also tried the only alternative for torch.cuda.empty_cache() in tensorflow which is to close the session and reset the graph I believe which didn’t work either.