# \[Core\] Task Status Check Failure in Ray Data Job with Preempted Workers

**URL:** <https://discuss.ray.io/t/core-task-status-check-failure-in-ray-data-job-with-preempted-workers/22343>\
**Category:** Ray Clusters\
**Created:** [April 21, 2025, 11:05am UTC](https://discuss.ray.io/t/core-task-status-check-failure-in-ray-data-job-with-preempted-workers/22343 "2025-04-21T11:05:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dragongu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dragongu/32/7243_2.png) [@dragongu](https://discuss.ray.io/u/dragongu)\
**Post date:** [April 21, 2025, 11:05am UTC](https://discuss.ray.io/t/core-task-status-check-failure-in-ray-data-job-with-preempted-workers/22343/1 "2025-04-21T11:05:03Z")

</div>

When running a simple Ray Data job where worker pods are frequently preempted, the following error occurs:

```auto
task_manager.cc:1412: Check failed: it->second.GetStatus() == rpc::TaskStatus::PENDING_NODE_ASSIGNMENT
task ID = 6ff6fe559f63b1b8b015cbfb3a695db0935ce25820000000 status = 1

```

It appears to be a multi-threaded state modification issue, but I haven’t located the exact line of code.

### Versions

2.43.0

---

<div class="post-metadata">

**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\
**Post date:** [April 22, 2025, 10:47pm UTC](https://discuss.ray.io/t/core-task-status-check-failure-in-ray-data-job-with-preempted-workers/22343/2 "2025-04-22T22:47:47Z")

</div>

Hi dragongu, do you have a way to recreate this possibly? There could be a few diff reasons this might be happening. Besides this error, are there any other errors happening? how is the resources (CPU / GPU / mem) for the tasks you’re running, like is it sufficient?

---

<div class="post-metadata">

**Author:** ![dragongu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dragongu/32/7243_2.png) [@dragongu](https://discuss.ray.io/u/dragongu)\
**Post date:** [April 23, 2025, 2:43am UTC](https://discuss.ray.io/t/core-task-status-check-failure-in-ray-data-job-with-preempted-workers/22343/3 "2025-04-23T02:43:22Z")

</div>

It is reproducible with abundant resources. For related issues, please refer to: [[Core] task\_manager.cc:1416: Check failed: it-\>second.GetStatus() == rpc::TaskStatus::PENDING\_NODE\_ASSIGNMENT · Issue #52530 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/52530)