How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. I am seeing the memory leakage while putting a object (numpy array) into object store via ray.put(). I am using ray==2.8.0 and python3.8 with ubuntu20.04. Please find attached the reproduction…

Please find some snapshots of memory utilisation from dashboard to understand memory consumption. [Screenshot from 2024-08-09 09-40-04] [Screenshot from 2024-08-09 12-19-05] [Screenshot from 2024-08-09 12-20-01] [Screenshot from 2024-08-09 12-20-51] [Screenshot from 2024-08-09 12-22-18] Pleas…

Does this repro after a ray upgrade to latest; on a new version of python and ubuntu 2204?

When you ray.put() you puts data into Ray object store. It’s kept there until the last reference to it is released. In your code I think the object is always held by frame → frame_holder → frame_holders, so it’s never released. I think this is WAI. If you release the references does it still leak?

@Ruiyang_Wang If I understand properly, you are saying that the object is being holded as frame_holder reference is stored in frame_holders. Can you please confirm this? And if you could explain, how exactly that could be the issue? Based on memory analysis, I could find followings: Object store …

Hi @Sam_Chan , I have run the script with ray=2.34.0 with python3.10 and ubuntu20.04. And, I could find the same behaviour of memory leakage with latest ray version also. Can you please suggest, how can I debug this part? Please let me know, if you need any extra information.

I wonder if it’s possible for you to not use ray.put? In your code: in Frame: just keep the numpy array in it as a field in PacketDecoder: put_async the frame with a numpy array to frame_holder in ODIActor: get_async from frame_holder and just use the inside numpy, no free. This way, Ray should b…

@Ruiyang_Wang Thanks for your suggestions. As per ray doc ( Anti-pattern: Passing the same large argument by value repeatedly harms performance — Ray 2.34.0 ), It would be anti pattern if your object is large and passing through multiple actors and you pass it directly without keeping it into object …

@Sam_Chan @Ruiyang_Wang I have generated git issue for the same. You can find it here: [Core] Memory leakage with ray.put() · Issue #47273 · ray-project/ray · GitHub . Please let me know if I can provide any other information.

TY for submitting - we’ll take a closer look next Monday.

Memory leakage with ray.put()

Ray Core

shyampatel August 13, 2024, 4:11am 9

I am getting the memory leakage in queue here also: Memory leakage with remote() calls.

Topic		Replies	Views
Memory leakage in object store memory Ray Core	4	459	August 9, 2024
Leaking worker memory Ray Core	9	569	February 19, 2021
Memory leak in ray Actor Ray Core	4	305	December 19, 2024
[Core] Having trouble evicting objects Ray Core	6	652	June 9, 2021
Object store spilling terabytes of data Ray Core	6	2785	January 11, 2023

Memory leakage with ray.put()

Related topics