# Driver on exit fails detached Actor Method

**URL:** https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134
**Category:** Uncategorized
**Created:** [July 5, 2024, 2:09am UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134 "2024-07-05T02:09:10Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![uduse](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/uduse/32/1524_2.png) [@uduse](https://discuss.ray.io/u/uduse)
#### Post date: [July 5, 2024, 2:09am UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134/1 "2024-07-05T02:09:10Z")

</div>

Say I have a script `run.py` locally that looks like

```python
from time import sleep
from uuid import uuid4
import ray

ray.init(...) # connect to my ray cluster, including all the runtime_env setups

@ray.remote
class LongTaskRunner:
    def run_long_task(self) -> None:
        print("Running long task")
        for i in range(120):
            print(f"Step {i + 1}/120")
            sleep(1)
        print("Task completed")

name = uuid4().hex
runner = LongTaskRunner.options(lifetime="detached", name=name).remote()
runner.run_long_task.remote()

sleep(5)

```

Then in my terminal I do

```auto
python run.py

```

Which will launch the actor in detached mode.

Though the Actor will remain alive after the driver exits, the call to Actor’s `run_long_task` will fail. According to the dashboard, the `ERROR STACK TRACE` says:

```auto
Error Type: WORKER_DIED

Job finishes (10000000) as driver exits. Marking all non-terminal tasks as failed.

```

How do I make sure the actor method don’t get terminiated?

---

<div class="post-metadata">

### Author: ![Ruiyang\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ruiyang_wang/32/4818_2.png) [@Ruiyang\_Wang](https://discuss.ray.io/u/Ruiyang_Wang)
#### Post date: [July 8, 2024, 4:55pm UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134/2 "2024-07-08T16:55:25Z")

</div>

Can’t reproduce. I am working in a ray cluster:

1. use your script to create a detached actor
2. use this script to get it and invoke another method

```auto
import ray

# namespace from the warning log
ray.init(namespace="f068f1e1-88cc-4dc5-9a1b-d1ab6ef6fe3a")

# name from Dashboard
a = ray.get_actor("e199350023f943b5a8cb696f8157f5aa")
ray.get(a.run_long_task.remote())

```

and it worked. In Dashboard I can see logs of both tasks from the 2 scripts, and after the task from this other script finished it returned from `ray.get` normally.

Can you make a repro?

---

<div class="post-metadata">

### Author: ![uduse](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/uduse/32/1524_2.png) [@uduse](https://discuss.ray.io/u/uduse)
#### Post date: [July 9, 2024, 2:26pm UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134/3 "2024-07-09T14:26:10Z")

</div>

What job submission interface are you using? I’m using Ray’s Python client, not `ray submit`.

---

<div class="post-metadata">

### Author: ![Ruiyang\_Wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ruiyang_wang/32/4818_2.png) [@Ruiyang\_Wang](https://discuss.ray.io/u/Ruiyang_Wang)
#### Post date: [July 9, 2024, 8:03pm UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134/4 "2024-07-09T20:03:50Z")

</div>

can you share your setup a bit more in detail? I created my local cluster via `ray.init()`

---

<div class="post-metadata">

### Author: ![uduse](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/uduse/32/1524_2.png) [@uduse](https://discuss.ray.io/u/uduse)
#### Post date: [July 10, 2024, 2:14pm UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134/5 "2024-07-10T14:14:43Z")

</div>

The script I shared is mostly comeplete. The only thing I hide is the address and other configuration in `ray.init`. I have a Ray cluster running remotely, and the `ray.init` just points to that cluster and sets up appropriate run environment.

In your experiment, you need to remove `ray.get` from `ray.get(a.run_long_task.remote())` . The whole idea is to dispatch the task without waiting it to complete, and hope the detached Actor keep doing the job even when the driver exits.

I suspect if you are using local ray, that ray dies with the driver anyways so detached Actor won’t be effective.

---

<div class="post-metadata">

### Author: ![Vibrat](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vibrat/32/7487_2.png) [@Vibrat](https://discuss.ray.io/u/Vibrat)
#### Post date: [December 28, 2024, 6:29am UTC](https://discuss.ray.io/t/driver-on-exit-fails-detached-actor-method/15134/6 "2024-12-28T06:29:08Z")

</div>

I experienced the same issue and turned out `ray job submit` will cancell tasks if the submitting process exits before all tasks are completed. Adding `--no-wait` will solve the issue.

```bash
ray job submit --no-wait --address http://localhost:8265 --working-dir . -- python remote3.py

```
