I’m training a fairly large model and am seeing this a lot: (pid=933) 2021-02-12 13:05:41,265 INFO trainable.py:72 -- Checkpoint size is 1446757015 bytes (pid=raylet) [2021-02-12 13:05:47,441 E 101 111] create_request_queue.cc:119: Not enough memory to create object fbee6bebb46e7ea09b9842a401000000…

Can Tune be configured to not keep paused trials in object store?

rliaw February 12, 2021, 6:40pm 5

Hmm, what type of search algorithm and scheduler are you using?

Topic		Replies	Views
Object Spilling useful to avoid running out of memory when using Ray Tune Ray Core	13	897	March 4, 2021
PPO Trial Runner:Process Events Pinned In Object Store RLlib	0	199	May 4, 2021
Running trials on GPU and the Object Storage Ray Tune	2	329	July 28, 2021
Ray Tune error resuming training from an AIR checkpoint Ray Libraries (Data, Train, Tune, Serve)	0	195	September 6, 2023
Checkpointing with distributed training Ray Tune	14	823	April 20, 2021