I’m training a fairly large model and am seeing this a lot: (pid=933) 2021-02-12 13:05:41,265 INFO trainable.py:72 -- Checkpoint size is 1446757015 bytes (pid=raylet) [2021-02-12 13:05:47,441 E 101 111] create_request_queue.cc:119: Not enough memory to create object fbee6bebb46e7ea09b9842a401000000…

Can Tune be configured to not keep paused trials in object store?

rliaw February 12, 2021, 6:40pm 5

Hmm, what type of search algorithm and scheduler are you using?

Topic		Replies	Views
Object Spilling useful to avoid running out of memory when using Ray Tune Ray Core	13	809	March 4, 2021
PPO Trial Runner:Process Events Pinned In Object Store RLlib	0	179	May 4, 2021
Ray Tune error resuming training from an AIR checkpoint Ray Libraries (Data, Train, Tune, Serve)	0	133	September 6, 2023
Checkpointing with distributed training Ray Tune	14	736	April 20, 2021
Terminated Trials hold memory causing OOM Ray Tune	1	363	October 26, 2021