An explicit thing I’m trying to do with hyper parameter search is to learn what the limits are with my 8gb of VRAM. I would like it so that if CUDA runs out of memory on a given run that this is simply treat as a high loss or some other “soft fail” that simply redirects the search algorithm somewher…

You’ll have to configure Tune to do that with a Search Algorithm! [image] Search Algorithms (tune.suggest) — Ray v1.1.0.dev0

How do I get tune.run to handle CUDA out of memory errors?

mm04926412 December 9, 2020, 11:44pm 12

Ah I see! I erroneously thought the schedueler handled this thank you!

1 Like

Topic		Replies	Views
RuntimeError: No CUDA GPUs are available Ray Tune	12	14713	February 3, 2023
Running out of CUDA Memory - Sample Batch , rollout fragment length is not helping RLlib	2	943	July 28, 2022
GPU memory not being freed every other trial in Ray Tune	3	712	February 21, 2023
How to specify max_calls for functional API Ray Tune	3	795	March 26, 2021
Ray Tune - CUDA OOM Error Ray Tune	0	503	July 26, 2021