The 4 “PINNED_IN_MEMORY” objects remain because the worker process (PID 256089) still holds deserialized references to them, even though all actors and tasks are dead. In Ray, objects are only released from the object store when all references—including those in worker memory—are gone. If a worker process does not exit cleanly or leaks references, these objects can remain pinned, preventing cleanup and blocking cluster downscaling (ba752da8, 624d3d09).
To resolve this, ensure the worker process (PID 256089) is terminated or restarts, which should release the references and allow Ray to clean up the objects and downscale the cluster. If the process is stuck, you may need to kill it manually. For robust production use, monitor for lingering worker processes and automate cleanup if necessary.
Would you like more detail on how to automate detection and cleanup of such leaks?
Sources:
- Memory Management (Ray 2.34.0)
- Proper workflow for keeping Ray memory clean and separating returned Python objects from their Ray references
Hint: Mention @RunLLM in the post for followups.