I was wondering if there’s some built-in feature to monitor the state of the submitted tasks to a
@ray.remote function. By “state” I mean if the task is either “pending”, “running” or “finished”. (similar to what is already being reported when using ray.tune)
My use case is quite simple: (1) a function submits N ray tasks (these will be scheduled and run in parallel just like ray’s core functionality); (2) however, I don’t have/want a
ray.get() blocking call that waits for these tasks to complete; (3) in other bits of my code I need to check at any given point how many of the submitted runs are “running”.
I cannot find a built-in functionality for this so this is how I have implemented it:
- I have an auxiliary Ray Actor that’s shared via an
ActorHandlewith each of the submitted tasks. The Actor class essentially behaves as a counter. In this way, when one of the submitted tasks starts running, it will increment by one the counter in the Actor. Before the tasks finished, i.e. before the
@ray.remotefunction exists, the counter is decremented by one. Below is the simplified code:
@ray.remote def task_launcher(task_id: int, monitor: ActorHandle): # increment counter and set task as "running" monitor.task_running.remote(task_id) # do stuff # decrement counter and set task as "finished" monitor.task_finished.remote(task_id)
- Having this Actor allows me to monitor how many tasks are running at a given point. This bit of information is passed to other functions in my code. (it can easily be extended to measure how many tasks have been completed and how many are still pending).
Am I re-inventing the wheel here? is this functionality already present?