We are using Ray Serve with Ray Operator 1.0.0 and Ray 2.10
We use container run time env and noticed that exited containers are not cleaned up. We ended up with an extremely large number of exited containers being stored and suspect that it is what led to this podman error Error: allocating lock for new container: allocation failed; exceeded num_locks (2048) and why new replicas couldn’t spin up. Once I manually removed the exited containers, new replicas were able to spin up.
Is there a configuration in runtime env for automatic clean up of exited containers and if not is there a plan to do so.
1 Like
Could be an issue; following up on [Serve] Exited containers not cleaned up · Issue #45093 · ray-project/ray · GitHub