This is a known Ray issue: sometimes, after a burst of task submissions, a subset of tasks can get permanently stuck in PENDING_NODE_ASSIGNMENT, even when resources are available and other identical tasks run and finish. This is not caused by resource shape, scheduling policy, autoscaling, or node state, and cancelling/resubmitting the stuck tasks does not help. The Ray scheduler and GCS do not mark these tasks as infeasible, and the tasks never get assigned to a node or worker, nor retried, despite cluster idleness. This bug has been reported by others and is not resolved by common workarounds or cluster restarts. See the detailed discussion and similar symptoms in this issue and this issue.
If you want more detail on root causes, debugging steps, or possible low-level mitigations, let me know.
Sources:
- https://discuss.ray.io/t/subset-of-tasks-stuck-in-pending-node-assignment-forever/9821
- https://github.com/ray-project/ray/issues/48996
Hint: Mention @Herald in the post for followups.