Hi @manuel.ramblr,
First of all, thank you for taking the time to share such detailed feedback on async inference and apologies for the delayed response.. Here are answers to your questions:
1. Heartbeats
Celery provides worker-level liveness checks, such as ping and worker heartbeat events, but it does not provide per-task heartbeats.
Celery supports a visibility_timeout. If a task is not acknowledged within this period, it becomes eligible for redelivery to another worker. The default varies by broker—for Redis, it is one hour. You can configure it to exceed the longest expected job duration:
CeleryAdapterConfig(
broker_url="redis://...",
backend_url="redis://...",
broker_transport_options={"visibility_timeout": 6 * 3600},
)
2. Consumer OOM and infinite retries
You are correct about the possibility of an indefinitely repeating failure loop. The current implementation uses Celery’s threads worker pool. If a task causes an out-of-memory error, the entire Celery worker process can terminate before it updates the retry counter. Once the visibility timeout expires, another worker may pick up the same task, encounter the same OOM condition, and repeat the cycle.
Unfortunately, this is a limitation of Celery’s threads pool in the current implementation. We expect to address this more effectively in the upcoming Taskiq task processor. For now, if you continue using Celery, an external retry counter or another durable mechanism would be required to prevent infinite redelivery.
3. Checkpointing and progress reporting
We agree that checkpointing and progress reporting would be valuable, particularly for long-running tasks. Neither capability is currently on our roadmap, but we would welcome a contribution in this area. If you decide to work on it, we would be happy to help you understand the implementation and guide you through the contribution process.
4. Idempotency
The current Celery task processor provides at-least-once delivery. As a result, a task may be executed more than once, and idempotency must currently be implemented at the application level.
We are actively working on a Taskiq-based task processor. Here’s the list of PRs on it. Please also continue sharing any other issues you encounter with Celery - this is exactly the kind of feedback we are looking for.
If Taskiq does not address your requirements and you decide to implement another task processor, you are welcome to contribute it directly to the ray repository. We would be glad to support you through that process.