1. Severity of the issue: (select one)
Medium: Significantly affects my productivity but can find a workaround.
2. Environment:
Ray version: 2.49.1
Python version: 3.10.18
OS: Ubuntu 24.04.3 LTS
Cloud/Infrastructure: Docker cluster
Other libs/tools (if relevant):
3. What happened vs. what you expected:
Expected: Ray keeps reference counts greater than 0 for all objects if they are required in later stages until all of them were utilized (e.g. an object created in one deployment should be kept in object store until another deployment successfully retrieves and uses it)
Actual: Certain objects reference count goes to zero and the object is being deleted, while still being requested later
I am working with a production system that heavily relies on Ray’s remote tasks and Ray’s Deployments. A majority of those deployments contains an ML/CV logic that produces somewhat large numpy arrays (hundreds, sometimes tens of MB for each array, usually tens of those are required for a successful run). We use the persistent Actor as an owner for all those arrays and we set the ownership with the NumpyStore wrapper like this:
Under normal conditions this ownership model works well, but under heavier load it occasionally fails at two specific spots, when calling retrieve method (so, essentially ray.get). I can simulate that heavy load by setting max_ongoing_requests at least to 3 and sending at least 2 requests to the system. Under those conditions I might get ReferenceCountingAssertionError, saying that a requested numpy array was deleted from memory while still being referenced. It is also worth noting that this error doesn’t appear consistently on the same data - even when running the same data over and over only a fraction of it will fail.
For know, unfortunately, I cannot create a short reproducible script. Inspecting ray memory output with RAY_record_ref_creation_sites=1doesn’t tell much; Ray logs with DEBUG verbosity only demonstrate that Ray honestly counts object references and deletes it when all possible counters drop to 0.
Here what I’ve already tried:
Getting rid of NumpyStore for those masks and returning/passing them as regular numpy arrays and corresponding references results in a severe memory leak. Main hypothesis here is that Ray workers become owners for those arrays and never die - therefore it keeps reference count above 0.
Moving to newer Ray (2.54.0) - didn’t help
Lowering and increasing kill_idle_workers_interval_ms and idle_worker_killing_time_threshold_ms parameters - didn’t correlate with error frequency
Lowering max_ongoing_requests - that lowers a simultaneous load and the error no longer reproduces.
My request for help sums up into 2 questions - Is our ownership model wrong and fragile, which leads to this error appearing? And how should one come up to debugging such errors? I know exactly where a faulty array is being created, what deployments and remote functions it passes through and where it might fail (same place, but unpredictable); it’s id in object store; even a worker/Deployment in which it was dereferenced.
Your ownership model—using a persistent actor as the owner via ray.put(data, _owner=actor)—is generally correct for keeping objects alive across deployments, but Ray’s distributed reference counting can still be fragile under high concurrency or complex object reference flows. The ReferenceCountingAssertionError indicates that, despite your efforts, the reference count for some objects is dropping to zero prematurely, causing Ray to delete the object before all consumers have finished using it. This can happen if references are not properly propagated or if intermediate tasks/actors holding references die or lose track of them before all downstream consumers have acquired their own references, especially under heavy load or parallel requests. This is a known challenge in Ray’s reference counting protocol, particularly when objects are passed through multiple tasks or actors and one of the intermediaries exits unexpectedly or fails to notify the owner about new borrowers (see discussion, reference counting doc).
For debugging, you are already using RAY_record_ref_creation_sites=1 and ray memory, which are the recommended tools. However, these tools may not always reveal subtle reference propagation issues, especially if the problem is due to lost notifications between actors or tasks. To further debug, ensure that:
All references to the object are explicitly passed and not just stored in local variables that may go out of scope.
No intermediate actor or task holding a reference is dying or being killed before all downstream consumers have acquired their references.
Avoid passing object references through multiple layers of tasks/actors if possible, or ensure that the owner is always notified of all borrowers (see this issue for a similar scenario).
Consider using named actors or a more centralized reference management pattern if you need strong guarantees.
Would you like a step-by-step breakdown of how Ray’s reference counting can fail in these scenarios, or more concrete suggestions for restructuring your workflow?
Certainly! Using named actors as a centralized reference manager means you create a long-lived, named actor (e.g., “PlasmaStoreActor”) whose sole responsibility is to store and manage all ObjectRefs that need to persist across tasks, actors, or deployments. Instead of passing ObjectRefs directly between tasks/actors (which can lead to lost references if an intermediate dies), you store all ObjectRefs in this actor and have all consumers retrieve the data via remote calls to the actor. This ensures the actor always holds a strong reference, preventing premature deletion, and provides a single place to manage object lifetimes and cleanup (see discussion, reference counting test).
Example pattern:
@ray.remote
class PlasmaStoreActor:
def __init__(self):
self.refs = {}
def put(self, key, data):
self.refs[key] = ray.put(data)
def get(self, key):
return ray.get(self.refs[key])
def delete(self, key):
del self.refs[key]
# Usage:
store = PlasmaStoreActor.options(name="PlasmaStoreActor", lifetime="detached").remote()
store.put.remote("my_array", np_array)
result = store.get.remote("my_array")
This approach avoids passing ObjectRefs through multiple layers and ensures the actor’s reference keeps the object alive until you explicitly delete it. It also makes debugging and memory management easier, as all references are tracked in one place (see also).
Would you like a step-by-step explanation of how this pattern prevents reference counting errors?
Is there another way? Introducing centralized reference manager would significantly complicated our code - not only we would have to manually put and delete new objects (numpy arrays), but also we would have to track which objects relates to which currently processed requests.
Another related question here - how does Ray internally understand when objects are no longer necessary and can be safely deleted? I understand how this works in simple setup (for example, a couple of tasks running one after another), but in our case we have a hierarchy of Ray Deployments, and created numpy arrays might be used really late in the code.
Ray uses distributed reference counting to track when objects can be safely deleted: an object remains in the object store as long as there is at least one live ObjectRef anywhere in the cluster, including references held by drivers, actors, tasks, or as arguments to pending tasks (Ray memory management docs). When all references go out of scope (i.e., are deleted or garbage collected), Ray decrements the count, and once it reaches zero, the object is eligible for eviction from the object store (reference counting doc).
In complex workflows with multiple deployments and late usage, Ray’s reference counting can become fragile—especially if ObjectRefs are passed through intermediate actors/tasks that may die or lose references before downstream consumers acquire their own. If you want to avoid a centralized manager, you must ensure that every consumer obtains its own ObjectRef directly from the original owner or that all intermediate holders remain alive until all consumers are done. Otherwise, you risk premature deletion. There is no built-in way to “pin” objects globally without manual management; Ray’s design relies on the lifetime of ObjectRefs in the cluster (see discussion, memory management FAQ).
Would you like a step-by-step example of how reference counting works in a multi-deployment scenario, or suggestions for minimizing manual tracking?