An explicit thing I’m trying to do with hyper parameter search is to learn what the limits are with my 8gb of VRAM. I would like it so that if CUDA runs out of memory on a given run that this is simply treat as a high loss or some other “soft fail” that simply redirects the search algorithm somewher…

You’ll have to configure Tune to do that with a Search Algorithm! [image] Search Algorithms (tune.suggest) — Ray v1.1.0.dev0

How do I get tune.run to handle CUDA out of memory errors?

rliaw December 9, 2020, 7:21pm 2

Hey @mm04926412, could you post the crash logs?

Usually, Tune shouldn’t fail overall if one single trial dies/

Topic		Replies	Views
RuntimeError: No CUDA GPUs are available Ray Tune	12	14695	February 3, 2023
Running out of CUDA Memory - Sample Batch , rollout fragment length is not helping RLlib	2	940	July 28, 2022
GPU memory not being freed every other trial in Ray Tune	3	712	February 21, 2023
How to specify max_calls for functional API Ray Tune	3	793	March 26, 2021
Ray Tune - CUDA OOM Error Ray Tune	0	502	July 26, 2021