[RLlib][Tune] Major memory leak 80GB (!) in 3 days (!)

[holinov]

I have a single node multi gpu ray cluser (v1.2.0)
I’m trainig an PPO agent an i’m having major memory leaks.
Trails on screenshot have nearly-same configuration (different hyper-parameters for reward function)
So you could see that some trails consume much more memory and memory consumption is constantly growing.
Just-started trail consumes only 1.2 GB

I could share some parts of my configuration if needed.
Does anyone know why this could happen and how to overcome it?

[sven1977]

Hey @holinov , thanks for the question. We did fix a leak recently (should be in 1.3 or in current master) in the SampleCollector thanks to @Bam4d .
Yeah could you post your config here as well? Or even better, a small reproduction script so we could debug this?

Thanks!