If a deployment returns an ObjectRef and the replica (owner) is downscaled or terminated, the object will be lost and you will get an OwnerDiedError when trying to access it. This is because Ray objects fate-share with their owner (the worker or actor that created them), and if the owner dies, the object is deleted from the object store (Ray fault tolerance docs, Ray Discourse). Using manual ray.put does not solve this, as the owner is still the process that called ray.put.
The proper way to pass objects between deployments in an autoscaling environment is to store the object with an owner that is guaranteed to persist, such as a detached actor. You can create a detached actor and use ray.put(data, _owner=actor_handle) to ensure the object fate-shares with the actor, not the ephemeral replica (Ray Discourse: object ownership, Ray Discourse: sharing objects with deployment). This way, as long as the detached actor is alive, the object will not be lost even if the original deployment is downscaled.
Would you like a code example of this pattern?
Sources:
- Ray fault tolerance docs
- Ray Discourse: sharing objects with deployment
- Ray Discourse: how to cache objects in the object store
- Ray Discourse: ray actor failover
Hint: Mention @RunLLM in the post for followups.