# \[Core\] Timeout individual remote tasks

**URL:** <https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844>\
**Category:** Ray Core\
**Created:** [April 22, 2021, 2:04pm UTC](https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844 "2021-04-22T14:04:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![eddie](https://avatars.discourse-cdn.com/v4/letter/e/f07891/32.png) [@eddie](https://discuss.ray.io/u/eddie)\
**Post date:** [April 22, 2021, 2:04pm UTC](https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844/1 "2021-04-22T14:04:27Z")

</div>

Consider submitting a collection of work to a remote method, where most tasks will complete in under a minute, but some may take orders of magnitude longer. The desire is to kill the outliers.

Assume that the number of tasks far exceeds the number of ray workers. This could be implemented by a user if, given an incomplete task\_id, there was a way to know if its status was queued vs assigned to a worker. Is there a way to do that?

Thanks, Eddie

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [June 3, 2021, 3:16pm UTC](https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844/2 "2021-06-03T15:16:45Z")

</div>

Hey @eddie , great question and sorry for the delay, which was caused by the question being “uncategorized”. It helps if you set a category (e.g. “RLlib”) when you post a new question. That way, we’ll find it more easily and can assign the right person to answer it.

Hey @Clark_Zinzow , could someone from the Ray core team answer this one here? Thanks 🙂

---

<div class="post-metadata">

**Author:** ![Clark\_Zinzow](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/clark_zinzow/32/445_2.png) [@Clark\_Zinzow](https://discuss.ray.io/u/Clark_Zinzow)\
**Post date:** [June 3, 2021, 9:02pm UTC](https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844/3 "2021-06-03T21:02:28Z")

</div>

Hi @eddie, unfortunately, there currently isn’t a way to set a per-task timeout or an API to determine if a task is queued or running. There’s an issue that was opened recently that contains a few different options, such as using the `ray.wait` API or using an actor to register, watch, and cancel tasks.

> <https://github.com/ray-project/ray/issues/15672>
>
> Hello @rkooo567 ,
> 
> Consider a scenario for which large numbers of tasks with s…ome outliers are present. We want the ability in ray in the form of python decorators to time-out individual tasks. Such functionality is provided in spark here: https://stackoverflow.com/questions/63389263/how-to-set-timeout-to-a-spark-task-or-map-operation-or-skip-long-running-task
> 
> Here is an example use case:
> 
> The below list user\_task consists of normal\_task and long\_task :
> user\_task \[normal\_task, normal\_task, normal\_task, long\_task, normal\_task, normal\_task\]
> 
> where normal\_tasks are tasks that run within time-out and long\_task is a task that runs for a very long time termed as an outlier.
> we looked at ray.wait and ray.cancel APIs it does not have the capability to set time-out at the individual task level, in other words, we want to provide an opportunity to run all the tasks in the above user\_tasks array and time-out outlier tasks.
> 
> Please let us know if you need any further details.

Please let us know if any of those patterns will work for you!

---

<div class="post-metadata">

**Author:** ![eddie](https://avatars.discourse-cdn.com/v4/letter/e/f07891/32.png) [@eddie](https://discuss.ray.io/u/eddie)\
**Post date:** [July 28, 2021, 2:42pm UTC](https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844/4 "2021-07-28T14:42:46Z")

</div>

Hi. Thanks for picking up this topic. I’ve gone thru the github enhancement discussion and don’t understand how it would work for a thousand tasks running with the number of workers dynamically changing between one worker and many. Also, it looks like asm582 was not successful getting the proposed actor-as-intermediary working.

Implementing something in the core is probably the simplest way to fix. Ray.wait() could then return 3 lists: completed, not-completed, timed-out.  
Thanks again for considering this issue.

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [July 28, 2021, 5:52pm UTC](https://discuss.ray.io/t/core-timeout-individual-remote-tasks/1844/5 "2021-07-28T17:52:57Z")

</div>

Thanks for bringing this up @eddie I brought this up again, and we will investigate what’s the best fix (you can see usability-hotfix label is added to the issue). You can follow up with the Github issue if things seem to go slowly!
