The spilled data from failed jobs is not automatically deleted because Ray does not currently implement disk-specific load balancing or automatic cleanup of spilled objects when jobs fail; spilled objects remain until their references are explicitly released or the node is restarted. This can lead to nodes running out of disk space, which in turn blocks new tasks from being scheduled, even if there are available CPUs, because Ray cannot create new objects or spill further when the disk is full. This behavior is confirmed in both user reports and Ray’s documentation, which note that Ray does not rebalance or clean up spilled objects across nodes and that disk pressure can block task scheduling even if other resources are available (e9bfd7ea, b6198811, 159bc4bb).
Tasks remain in the PENDING_NODE_ASSIGNMENT state because, with the disk full, Ray cannot create or spill new objects required for task execution, effectively blocking scheduling on those nodes. Even if CPUs are available, Ray’s scheduler will not assign tasks if the node cannot handle object creation due to disk pressure (b6198811, e9bfd7ea). To resolve this, you should manually clean up the spill directories on the affected nodes or restart the nodes to free up disk space. For future runs, consider increasing disk size or tuning your workload to reduce object store and spill usage.
Would you like more detail on how to monitor or clean up spilled objects, or on configuring Ray to better handle large workloads?
Sources:
- Ray cannot handle out of disk space even when other nodes have available disk space
- Unexpected OutOfDiskError caused by Full Local Object Store
- 159bc4bb
Hint: Mention @RunLLM in the post for followups.