Subset of tasks permanently stuck in PENDING_NODE_ASSIGNMENT with the cluster fully idle

Ray Core
1

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.

2. Environment:

  • Ray version: 2.54.0
  • Python version: 3.12
  • OS: Ubuntu 24.04
  • Cloud/Infrastructure: Hetzner cloud
  • Other libs/tools (if relevant):

3. What happened vs. what you expected:

  • Expected:
    Ray tasks get dispatched (move from PENDING_NODE_ASSIGNMENT to RUNNING or similar) when cluster resources are available, and all tasks in the job get attempted.

  • Actual:
    Running a batch workload that dispatches ~250-450 near-identical @ray.remote tasks from a single driver in one burst, a subset of those tasks get stuck permanently in PENDING_NODE_ASSIGNMENT and are never scheduled — while hundreds of sibling tasks with identical resource requirements, submitted around the same time, get scheduled normally, run, and complete.

    The affected tasks:

    • Show node_id: None, worker_id: None, attempt_number: 0 in ray.util.state.list_tasks() output — never assigned a node, never retried, not even once.
    • Have required_resources, runtime_env_info, and argument sizes indistinguishable from sibling tasks that schedule fine (see “What we ruled out”).
    • Remain stuck for the entire remaining duration of a run (many hours), even as cluster-wide CPU utilization goes to 0.
    • Are never flagged by Ray’s own infeasibility tracking (Infeasible queue length: 0 in raylet state dumps throughout).

4. Attempted workaround

We built an application-level mitigation: detect a task that’s been PENDING_NODE_ASSIGNMENT for >90s while a node genuinely has enough free CPU for it (computed per-node, not from ray.available_resources()'s cluster-wide sum — see below), then ray.cancel(ref, force=True) it and immediately resubmit via .remote() again, which gives it a brand-new task_id. This does not help. In one production run, a specific set of ~20 tasks (identifiable by their arguments, which encode a stable identity — see next section) were cancelled and resubmitted repeatedly.

5. Things we we ruled out

  • Not SPREAD-specific. tested scheduling_strategy="SPREAD" vs default hybrid policy in production. Both produced the same pattern.
  • Not resource-shape specific. Tested {CPU: 10, memory: 12GiB} vs {CPU: 10} only. Both hit the identical bug.
  • Not tied to the pending-task submission cap. Tried limiting the number of dispatched tasks to at most 150, 200, and 500. All produced the same results.
  • Not accumulated GCS/raylet state. Fully restarted the cluster (new GCS process, new node IDs across all 20 nodes) between occurrences. The bug reproduced again on the fresh cluster.
  • Not duplicate/misconfigured node IPs. Checked ray.nodes()[*].NodeManagerAddress across all 20 nodes; all unique.
  • Not a custom-resource or env-var override. All 19 worker nodes report identical CPU=32.0, memory=126.4GB, zero custom resources, no RAY_* scheduler-related env var overrides.
  • Not a worker/node crash. Checked ray.util.state.list_tasks() for WORKER_DIED or any error-classified task in the affected job’s actual execution window — zero.
  • Not large task arguments. Checked ray.util.state.list_objects() for anything associated with these tasks exceeding ~500KB; found nothing — arguments should be well within the normal inlining path.
  • Not an unusual resource request. required_resources and runtime_env_info for stuck tasks (via list_tasks(detail=True)) are byte-for-byte identical in shape to tasks that scheduled fine ({'CPU': 10.0}, same runtime_env_info, no placement_group_id, no label_selector).
  • No raylet-level signal. Grepped the head node’s raylet log for spillback, lease request, and infeasible around the creation-time window of stuck tasks. Zero matches outside routine periodic state-dump headers.
  • No GCS-level signal. Same result searching gcs_server.out.
  • Autoscaling is not a factor. Cluster uses the Ray readonly autoscaler provider (max_workers: 0), a static externally-provisioned pool; no node-launch dynamics apply.
2

This is a known Ray issue: sometimes, after a burst of task submissions, a subset of tasks can get permanently stuck in PENDING_NODE_ASSIGNMENT, even when resources are available and other identical tasks run and finish. This is not caused by resource shape, scheduling policy, autoscaling, or node state, and cancelling/resubmitting the stuck tasks does not help. The Ray scheduler and GCS do not mark these tasks as infeasible, and the tasks never get assigned to a node or worker, nor retried, despite cluster idleness. This bug has been reported by others and is not resolved by common workarounds or cluster restarts. See the detailed discussion and similar symptoms in this issue and this issue.

If you want more detail on root causes, debugging steps, or possible low-level mitigations, let me know.

Sources:

Hint: Mention @Herald in the post for followups.

3

Yeah give me more detail on root causes and possible low-level mitigations.

4

@lucas-at-arete Thank you for reporting. This sounds like a bug in a very basic task use case. Could you provide a minimal reproducible script? I am happy to take a deep dive.

5

@Yicheng_Lu thanks for your response. To be honest i’m struggling to create a minimal reproducible example…

6

Follow-up with some new diagnostic angles (Ray 2.58.0)

Hit the same bug on a fresh cluster (10 worker nodes × 32 CPU, Hetzner-hosted, Dockerized raylets, CPU-only tasks at num_cpus=10, scheduling_strategy="SPREAD" ). 47 of 420 tasks dispatched in a single burst at job start got permanently wedged in PENDING_NODE_ASSIGNMENT, cluster otherwise ~65% idle. Posting what we found in case it narrows the search:

Confirmed / ruled out:

  • No dead or stale nodes anywhere — ray.nodes(), ray status, and every raylet’s own ClusterResourceManager view agreed on exactly 10 real, alive workers the entire time. Grepped gcs_server.out/raylet.out for the full session for any node-removal event — zero.
  • No broken TCP. Cross-referenced the driver’s open FDs against /proc/net/tcp6 and found one clean ESTABLISHED connection to every single worker, all with <20ms RTT and near-zero retransmits.

New finding — the server side is stuck too, but its own scheduler queue is empty:
On every worker’s debug_state.txt, NodeManagerService.grpc_server.RequestWorkerLease showed a nonzero “active” count (3–6 per node) that had been stuck for hours. Summed across all 10 workers: exactly 47 — a 1:1 match with the driver’s 47 stuck grpc_client.RequestWorkerLease calls. But every one of those same workers’ ClusterLeaseManager section reads Infeasible queue length: 0, Schedule queue length: 0, Grant queue length: 0, num_waiting_for_resource: 0 — on all 10 nodes, no exceptions. The scheduler believes there’s nothing outstanding; the RPC-handler-level “active” stat disagrees, forever.

No thread is actually stuck in the handler: installed gdb inside two different worker containers and did thread apply all bt across 86 threads per raylet, 172 total. Zero threads on either node were executing inside RequestWorkerLease/PopWorker/HandleWorkerAvailable — every thread was idle in a gRPC poller/event-loop wait.

Our read: this points at a lost/leaked completion callback rather than a hang, deadlock, or dead-peer scenario — the scheduling decision completes and releases its resources cleanly, but whatever step sends the gRPC reply (and decrements that handler’s “active” stat) never fires for a small subset of requests.

Timeline oddity: the driver’s per-minute event-stats dumps show the stuck-lease count wasn’t perfectly static — held at 58 for 15.5 hours, then drained to 47 over a 23-minute window with no corresponding node event or anything unusual in the logs, flat at 47 since.

Worker start/disconnect churn was our leading candidate for what triggers the lost callback — now falsified. We noticed the original incident’s burst dispatch caused a mass worker-start/mass-disconnect storm on every node in the first ~12 seconds (up to 32 workers started, 29–54 immediately disconnected as excess), and initially took that as the likely cause. Live reproduction says otherwise: escalating synthetic bursts (40 → 150 → 200 → 400 tasks) at the cluster’s idle capacity, including a 400-task, all-10-node burst under scheduling_strategy="SPREAD" with idle pools cleared first, produced far more churn than the original (406 workers started and disconnected on the busiest node alone) and reproduced zero stuck leases. So the churn correlation we saw was most likely incidental, not causal — the mechanism (lost callback) still stands, but the trigger remains open. The one condition we didn’t test is a genuinely cold session (this cluster’s/session’s first-ever job).

Happy to share raw debug_state.txt/raylet.out snippets or the (unsuccessful) repro scripts if useful to anyone interested in working on this.

7

Follow-up with a tentative workaround

I was able to free up those stuck tasks by killing worker nodes. Upon killing off all of the completely idle nodes, the stuck tasks were queued and eventually completed. I currently have a watchdog script in place that polls the head node for resource usage and kills off completely idle nodes 1 at a time. The idea being that this should free stuck tasks, and downsize the cluster at the same time. TBD on if is a robust solution…