Async workflow patterns in Ray

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi folks, what’s a good pattern for creating async workflows (submit a job to a pool of workers, poll for completion, retrieve results) in ray? I want to use ray actors and ray tasks, but the issue is that ray actors don’t autoscale as far as I can see
I think it would be inefficient to have to re-create the runtime environment every time if I want to run the same async operation on a bunch of things, so I can’t use independent ray jobs
Ray batch is an option but am looking to process things as we ingest them, rather than scheduling a batch job every X hours

Have you tried detached actors? You don’t need to recreate the actors the next time you call ray.init.