I’m training a fairly large model and am seeing this a lot: (pid=933) 2021-02-12 13:05:41,265 INFO trainable.py:72 -- Checkpoint size is 1446757015 bytes (pid=raylet) [2021-02-12 13:05:47,441 E 101 111] create_request_queue.cc:119: Not enough memory to create object fbee6bebb46e7ea09b9842a401000000…

To follow up - it appears that I should be concerned about this because while it recovered several times, the tuning job eventually failed due to this error.

[image] matthew.cox: 1446757015 Hmm, how many concurrent trials are you running?

I haven’t explicitly capped concurrent trials - I’ve got a max of 36 trials and due to resource constraints, only 13 can actually be running at once. Do you think that setting max_concurrent may help per User Guide & Configuring Tune — Ray v2.0.0.dev0 ?

Hmm, I see… yeah I guess object spilling should be the right way to do this.

Can Tune be configured to not keep paused trials in object store?

rliaw February 12, 2021, 6:40pm 5

Hmm, what type of search algorithm and scheduler are you using?

Topic		Replies	Views
Object Spilling useful to avoid running out of memory when using Ray Tune Ray Core	13	1000	March 4, 2021
PPO Trial Runner:Process Events Pinned In Object Store RLlib	0	212	May 4, 2021
Ray using so much memory I cannot even start the tuning Ray Tune	5	2499	April 24, 2023
Ray Tune is Frozen, Large Number of Trials are Paused Ray Tune	1	471	October 26, 2021
Most runs immediately failing with "out of memory" Ray Tune	5	1306	May 11, 2021

Can Tune be configured to not keep paused trials in object store?

Related topics