I have found object spilling to be useful to help avoid running out of memory when training a large model. I’m using Ray Tune to tune hyperparameters for a model that has about 4.5GB worth of parameters. For some reason, whenever a trial is paused, the trial is saved in the object store, and is neve…

Object Spilling useful to avoid running out of memory when using Ray Tune

matthew.cox February 25, 2021, 6:23pm 4

Topic		Replies	Views
Can Tune be configured to not keep paused trials in object store?	6	526	February 15, 2021
(raylet)file_system_monitor.cc:105: - "Object creation will fail if spilling is required" Ray Tune	5	4693	September 9, 2022
Ray using so much memory I cannot even start the tuning Ray Tune	5	1238	April 24, 2023
How to stop ray tune running out of space on disk Ray Libraries (Data, Train, Tune, Serve)	3	456	November 4, 2022
Most runs immediately failing with "out of memory" Ray Tune	5	903	May 11, 2021