# Monitor state of submitted Tasks

**URL:** <https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159>\
**Category:** Ray Core\
**Created:** [May 14, 2021, 2:22pm UTC](https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159 "2021-05-14T14:22:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jafermarq](https://avatars.discourse-cdn.com/v4/letter/j/dec6dc/32.png) [@jafermarq](https://discuss.ray.io/u/jafermarq)\
**Post date:** [May 14, 2021, 2:22pm UTC](https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159/1 "2021-05-14T14:22:38Z")

</div>

Hi,

I was wondering if there’s some built-in feature to monitor the state of the submitted tasks to a `@ray.remote` function. By “state” I mean if the task is either “pending”, “running” or “finished”. (similar to what is already being reported when using ray.tune)

My use case is quite simple: (1) a function submits N ray tasks (these will be scheduled and run in parallel just like ray’s core functionality); (2) however, I don’t have/want a `ray.get()` blocking call that waits for these tasks to complete; (3) in other bits of my code I need to check at any given point how many of the submitted runs are “running”.

I cannot find a built-in functionality for this so this is how I have implemented it:

- I have an auxiliary Ray Actor that’s shared via an `ActorHandle` with each of the submitted tasks. The Actor class essentially behaves as a counter. In this way, when one of the submitted tasks starts running, it will increment by one the counter in the Actor. Before the tasks finished, i.e. before the `@ray.remote` function exists, the counter is decremented by one. Below is the simplified code:

```auto
@ray.remote
def task_launcher(task_id: int, monitor: ActorHandle):

    # increment counter and set task as "running"
    monitor.task_running.remote(task_id)

    # do stuff

    # decrement counter and set task as "finished"
    monitor.task_finished.remote(task_id)

```

- Having this Actor allows me to monitor how many tasks are running at a given point. This bit of information is passed to other functions in my code. (it can easily be extended to measure how many tasks have been completed and how many are still pending).

Am I re-inventing the wheel here? is this functionality already present?

---

<div class="post-metadata">

**Author:** ![Mingwei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mingwei/32/997_2.png) [@Mingwei](https://discuss.ray.io/u/Mingwei)\
**Post date:** [May 14, 2021, 4:26pm UTC](https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159/2 "2021-05-14T16:26:51Z")

</div>

Can `ray.wait([...])` meet the use case?

---

<div class="post-metadata">

**Author:** ![jafermarq](https://avatars.discourse-cdn.com/v4/letter/j/dec6dc/32.png) [@jafermarq](https://discuss.ray.io/u/jafermarq)\
**Post date:** [May 14, 2021, 4:39pm UTC](https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159/3 "2021-05-14T16:39:53Z")

</div>

Good point. I was using `ray.wait()` before but, since it only returns the lists of “completed” and “uncompleted” objects, it requires extra logic to determine how many of the “uncompleted” are currently running. I found the approach of having an `Actor` more convenient… but still requires a few lines of code just to have this simple functionality working. This is why I was wondering if something simpler, already in the Ray framework exists.

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [May 14, 2021, 5:32pm UTC](https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159/4 "2021-05-14T17:32:02Z")

</div>

Hey @jafermarq. Thanks for asking this. Your approach is definitely a working approach, and I will do the same thing if I’d like to get more detailed information about each task. You can even improve the observability by tracking of other metrics like running time or memory usage.

Unfortunately, there’s no built-in feature for task stats. We tried implementing it in the past, and the it was somewhat deprioritized. If you are interested in the built-in feature, it’ll be great if you start a feature request, so the team will see the user demand (which means we will more likely to spend resources on it)!

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [May 14, 2021, 6:32pm UTC](https://discuss.ray.io/t/monitor-state-of-submitted-tasks/2159/5 "2021-05-14T18:32:22Z")

</div>

plus 1 on this request, I think it will help me with the issue raised here:

> <https://github.com/ray-project/ray/issues/15672>
>
> Hello @rkooo567 ,
> 
> Consider a scenario for which large numbers of tasks with s…ome outliers are present. We want the ability in ray in the form of python decorators to time-out individual tasks. Such functionality is provided in spark here: https://stackoverflow.com/questions/63389263/how-to-set-timeout-to-a-spark-task-or-map-operation-or-skip-long-running-task
> 
> Here is an example use case:
> 
> The below list user\_task consists of normal\_task and long\_task :
> user\_task \[normal\_task, normal\_task, normal\_task, long\_task, normal\_task, normal\_task\]
> 
> where normal\_tasks are tasks that run within time-out and long\_task is a task that runs for a very long time termed as an outlier.
> we looked at ray.wait and ray.cancel APIs it does not have the capability to set time-out at the individual task level, in other words, we want to provide an opportunity to run all the tasks in the above user\_tasks array and time-out outlier tasks.
> 
> Please let us know if you need any further details.