PPO with PyTorch GPU has a RAM memory leak for Ray 1.6.0

Hi,

I am seeing a steady increase in RAM usage while training with multi-agent PPO on a GPU with Ray 1.6. The GPU memory usage is stable.
I am also running the same training on CPU with Ray 1.0.0 and have no memory issues.

Any ideas for a possible solution will be appreciated.

Thank you,
priyam

Seen similar leak with Ray 1.6 Torch SAC with replay buffer set to 500k:
ray_torch_sac_500K_buffer_leak

Eventually I get (15 GB machine):

PID    MEM    COMMAND
11076  7.48GiB ray::SAC.train()
11077  4.34GiB ray::RolloutWorker

Are you also using a GPU?

Are you also using a GPU?

Yes.