How to trigger callback in driver program after parallel process finishes executing?

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I have post-processing steps that I need to execute asynchronously in the driver program for context reasons. I know multiprocessing Pool “apply_async” allows you to specify a callback function. Is something like this possible with Ray? I know about “ray.wait()” but I’m not sure how to use it in a non-blocking fashion to trigger callbacks as completed IDs become available.

Thanks!
Alec

2

The pattern I ended up using successfully is:

def _ray_getter(i):
    try:
        _callback_fn(ray.get(i))
    except Exception as e:
        _error_callback_fn(e)

thread = threading.Thread(target=_ray_getter,args=(func.remote(*args),))
thread.start()