# Subset of tasks permanently stuck in PENDING\_NODE\_ASSIGNMENT with the cluster fully idle **URL:** \ **Category:** Ray Core\ **Created:** [September 10, 2026, 5:04pm UTC](https://discuss.ray.io/t/subset-of-tasks-permanently-stuck-in-pending-node-assignment-with-the-cluster-fully-idle/23616 "2026-09-10T17:04:27Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\ **Post date:** [September 10, 2026, 5:04pm UTC](https://discuss.ray.io/t/subset-of-tasks-permanently-stuck-in-pending-node-assignment-with-the-cluster-fully-idle/23616/2 "2026-09-10T17:04:47Z")
This is a known Ray issue: sometimes, after a burst of task submissions, a subset of tasks can get permanently stuck in PENDING\_NODE\_ASSIGNMENT, even when resources are available and other identical tasks run and finish. This is not caused by resource shape, scheduling policy, autoscaling, or node state, and cancelling/resubmitting the stuck tasks does not help. The Ray scheduler and GCS do not mark these tasks as infeasible, and the tasks never get assigned to a node or worker, nor retried, despite cluster idleness. This bug has been reported by others and is not resolved by common workarounds or cluster restarts. See the detailed discussion and similar symptoms in [this issue](https://discuss.ray.io/t/subset-of-tasks-stuck-in-pending-node-assignment-forever/9821) and [this issue](https://github.com/ray-project/ray/issues/48996). If you want more detail on root causes, debugging steps, or possible low-level mitigations, let me know. Sources: - [https://discuss.ray.io/t/subset-of-tasks-stuck-in-pending-node-assignment-forever/9821](https://discuss.ray.io/t/subset-of-tasks-stuck-in-pending-node-assignment-forever/9821) - [https://github.com/ray-project/ray/issues/48996](https://github.com/ray-project/ray/issues/48996) Hint: Mention @Herald in the post for followups. --- _[View the full topic](https://discuss.ray.io/t/subset-of-tasks-permanently-stuck-in-pending-node-assignment-with-the-cluster-fully-idle/23616)._