# Seeing weirdness when running Ray Dag

**URL:** <https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927>\
**Category:** Ray Core\
**Created:** [August 24, 2023, 2:21pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927 "2023-08-24T14:21:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wub](https://avatars.discourse-cdn.com/v4/letter/w/e56c9b/32.png) [@wub](https://discuss.ray.io/u/wub)\
**Post date:** [August 24, 2023, 2:21pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927/1 "2023-08-24T14:21:09Z")

</div>

If I bind all my dag node before submitting my script to Ray, why would I see new task getting added after ~100 tasks finish?

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [August 24, 2023, 2:34pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927/2 "2023-08-24T14:34:26Z")

</div>

Can you share a script that represents the scenario? I don’t think I understood the question.

---

<div class="post-metadata">

**Author:** ![wub](https://avatars.discourse-cdn.com/v4/letter/w/e56c9b/32.png) [@wub](https://discuss.ray.io/u/wub)\
**Post date:** [August 24, 2023, 4:24pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927/3 "2023-08-24T16:24:31Z")

</div>

It is a bit difficult to share a minimal reproducible script - I will still give it a try but lemme describe what I am seeing in detail first.

Say I have a DAG, which has 167 tasks in total. When submitting to Ray, what I find is e.g. 130 tasks are shown in the dashboard (these 130 finish really fast). Then, I am starting to see the job in a “stuck” stage, after a while, the 131st task will appear and gets run. Eventually after a long time, all 167 tasks will finish but this looks off.

Ray 2.6.1  
Python 3.10.12

---

<div class="post-metadata">

**Author:** ![wub](https://avatars.discourse-cdn.com/v4/letter/w/e56c9b/32.png) [@wub](https://discuss.ray.io/u/wub)\
**Post date:** [August 24, 2023, 4:31pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927/4 "2023-08-24T16:31:11Z")

</div>

A somewhat related q:  
If I run:

```auto
import ray

@ray.remote
def calc1(x,y):
    return x+y

@ray.remote 
def calc2(x):
    return x+1

dts = range(100)
outputs = {}

for _dt in dts:
    outputs[_dt] = calc2.bind(calc1.bind(1,2))

@ray.remote
def join_for_dag(*args):
    return 

tasks = [out for out in outputs.values()]

final = join_for_dag.bind(tasks)

print(ray.get(final.execute()))

```

Why would I see the job marked as SUCCEEDED on the dashboard but with 190 tasks out of 201 actually failed?

---

<div class="post-metadata">

**Author:** ![wub](https://avatars.discourse-cdn.com/v4/letter/w/e56c9b/32.png) [@wub](https://discuss.ray.io/u/wub)\
**Post date:** [August 24, 2023, 7:17pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927/5 "2023-08-24T19:17:06Z")

</div>

re: the original question, I find my job runs perfectly after I switch from .bind to .remote. Might be a bug in ray dag.

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [August 25, 2023, 1:57pm UTC](https://discuss.ray.io/t/seeing-weirdness-when-running-ray-dag/11927/6 "2023-08-25T13:57:59Z")

</div>

Let me try repro next week!