You’re correct—simply adding more storage may only delay the problem if the underlying causes (object leaks, unbounded spilling, or lack of cleanup) aren’t addressed. The best practice is to size storage so that the working set and expected spill volume fit comfortably, but also to monitor and control memory usage, spill rates, and reference lifetimes (ba752da8, 7094df69).
For robust production, combine increased storage with:
- Monitoring object store and disk usage.
- Limiting the number of concurrent jobs/tasks.
- Ensuring all references are released after job completion.
- Periodic cleanup or node restarts to clear leaks.
Would you like more detail on how to estimate storage needs or set up monitoring and cleanup?
Sources:
Hint: Mention @RunLLM in the post for followups.