I have found object spilling to be useful to help avoid running out of memory when training a large model. I’m using Ray Tune to tune hyperparameters for a model that has about 4.5GB worth of parameters. For some reason, whenever a trial is paused, the trial is saved in the object store, and is neve…

Object Spilling useful to avoid running out of memory when using Ray Tune

matthew.cox February 25, 2021, 6:23pm 4

Topic		Replies	Views
Object Spilling Ray Tune	1	725	January 10, 2023
Can Tune be configured to not keep paused trials in object store?	6	575	February 15, 2021
Ray Spilled the object even though lots of /dev/shm is empty Ray Core	6	311	September 29, 2023
(raylet)file_system_monitor.cc:105: - "Object creation will fail if spilling is required" Ray Tune	5	5816	September 9, 2022
Ray using so much memory I cannot even start the tuning Ray Tune	5	1990	April 24, 2023