Severity: pretty hard
Hi all,
As I’m discovering about Ray, I would love to be able to integrate it with my Django application as it appears to be a much more powerful and flexible option than Celery, especially for my needs.
However, I am noticing that the way the Python “client” works seems to be conceputally incompatible. Indeed, it appears that you should call ray.init(), which will connect the current driver to the cluster and start a job. This job will then continue until being stopped — potentially months later, as it is a long-running web server.
On the contrary, if the job gets finished early then all subsequent tasks appear to be terminated, but that’s not what I want: the goal is to start tasks on my Ray cluster in a fire-and-forget mode, the tasks usually being the ones in charge of notifying of their own completion — either by changing a row in the database and/or by broadcasting a message to websocket consumers.
You’ll understand that overall, for use from a web server, the behavior of Celery is what I wish.
Now after digging into the code, I discover there a global_worker and _global_node, which are hardwired into the decorators in a way that makes it impossible to scope those to the current context (whichever this might be).
The closest thing that I’m finding is to use JobSubmissionClient, however this in turns looks excessively tedious.
As a result, I’m kind of stuck with no good pattern nor way to manage this issue. As I’m looking into it I’ll be sure to report my findings, however if anyone else encounters interesting results, I’m all ears.
Thanks
Rémy
