# \[RLlib\] GPU Memory Leak? Tune + PPO, Policy Server + Client

**URL:** <https://discuss.ray.io/t/rllib-gpu-memory-leak-tune-ppo-policy-server-client/10763>\
**Category:** RLlib\
**Created:** [May 23, 2023, 8:53pm UTC](https://discuss.ray.io/t/rllib-gpu-memory-leak-tune-ppo-policy-server-client/10763 "2023-05-23T20:53:42Z")\
**Posts on this page:** 1\
**Showing post:** 10

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [May 24, 2023, 9:07pm UTC](https://discuss.ray.io/t/rllib-gpu-memory-leak-tune-ppo-policy-server-client/10763/10 "2023-05-24T21:07:05Z")

</div>

Hi @Denys_Ashikhin and @kourosh,

Here are two posts that might be relevant here.

> [@Memory issue debugging](https://discuss.ray.io/t/memory-issue-debugging/7573/4):
>
> Hi @Blubberblub, My guess is that your environment is generating samples much faster than the training is consuming them. This is causing the sample queue to fill up in the policy server which is causing the memory issue. Try adding the following print and see if this is growing: @override(InputReader) def next(self): print(f"Size of samples queue is: {self.samples\_queue.qsize()}", ) return self.samples\_queue.get() My env is not much faster than the training but if I…

> <https://github.com/ray-project/ray/issues/25615#issuecomment-1434823961>
>
> \### What happened + What you expected to happen
> 
> I was hoping that an RLlib tr…aining script could hold a roughly constant amount of memory after at most 1M environment interactions. But the memory usage grows unboundedly, which prohibits long-running training, especially using \`ray.tune\` to train multiple experiments concurrently. 
> 
> The memory leak happens with \`'framework': 'tf2'\`. \`'framework': 'tf'\` seems fine, but our custom algorithm is targeting TensorFlow 2 for the "tracing" feature. 
> 
> The memory leak rate is around 1% RAM every 5M environment interactions (on a 64G RAM machine). It should be noted that we use the nightly Ray version because the fixes in https://github.com/ray-project/ray/pull/15412 are not released yet. Before this fix (e.g. Ray v1.12.1) leaks at a much faster rate of 1% RAM every 1M environment interactions.
> 
> We tried to debug with the help of \`MemoryTrackingCallbacks\`, but shows no obvious leaks (This is different from before https://github.com/ray-project/ray/pull/15412, where the callback shows memory allocated for global\_timestep is constantly increasing).
> 
> !\[image\](https://user-images.githubusercontent.com/13808089/172759068-4cb1859b-dae4-4377-9ca1-8a92d7b54ab2.png)
> 
> 
> \### Versions / Dependencies
> 
> Ray: Daily Release at 2022-06-09T02:28:12.704Z from 'https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp39-cp39-manylinux2014\_x86\_64.whl'
> 
> TensorFlow: 2.9.1 CPU
> 
> Python: Python 3.9.13
> 
> OS: Ubuntu 18.04.2 LTS
> 
> 
> \### Reproduction script
> 
> \#### Usage
> \`\`\`bash
> \# Installation
> conda create -n rllib-leak -c conda-forge --override-channels python=3.9
> conda activate rllib-leak
> pip install tensorflow 'ray\[rllib\] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp39-cp39-manylinux2014\_x86\_64.whl'
> \# Run
> python ppo\_leak.py
> \`\`\`
> Then TensorBoard's \`ray/tune/perf/ram\_util\_percent\` or \`ray/tune/custom\_metrics/tracemalloc/worker/rss\_mean\` (if MemoryTrackingCallbacks is used) shows constant memory increment.
> 
> \#### Dummy Env (dummy\_env.py)
> \`\`\`python
> from gym import Env
> from gym.spaces import Box
> from gym.wrappers import TimeLimit
> import numpy as np
> 
> 
> class DummyEnv(Env):
> observation\_space = Box(-np.inf, np.inf, (70,))
> action\_space = Box(-1.0, 1.0, (12,))
> 
> \_dummy\_obs = np.random.rand(70)
> 
> def step(self, action):
> return DummyEnv.\_dummy\_obs, 0.0, False, {}
>     
> def reset(self):
> return DummyEnv.\_dummy\_obs
> 
> def env\_creator(config):
> return TimeLimit(DummyEnv(), max\_episode\_steps=1000)
> \`\`\`
> 
> \#### Script (ppo\_leak.py)
> \`\`\`python
> from ray.rllib.agents.callbacks import MemoryTrackingCallbacks
> from ray.rllib.agents.ppo import PPOTrainer
> 
> import ray.tune.registry
> import dummy\_env
> ray.tune.registry.register\_env('dummy', dummy\_env.env\_creator)
> 
> agent\_config = {
> # 'callbacks': MemoryTrackingCallbacks,
> 
> # Env config
> 'env': 'dummy',
> 
> # Worker config
> 'framework': 'tf2',
> 'eager\_tracing': True,
> 'num\_workers': 10,
> 'num\_envs\_per\_worker': 4,
> 'num\_gpus': 0,
> 'horizon': 1000,
> 
> # PPO config
> 'rollout\_fragment\_length': 1000,
> 'train\_batch\_size': 40000,
> 'sgd\_minibatch\_size': 40000,
> 'num\_sgd\_iter': 30,
> 'lr': 3e-4,
> 'lambda': 0.97,
> 'gamma': 0.99,
> 'no\_done\_at\_end': True,
> 
> # Model config
> 'model': {
> '\_use\_default\_native\_models': False, # Both True/False leaks at same rate
> 'fcnet\_hiddens': \[256, 256\],
> 'fcnet\_activation': 'relu',
> 'vf\_share\_layers': False,
> 'free\_log\_std': True,
> },
> "seed": 1,
> }
> 
> trainer = PPOTrainer(agent\_config)
> while True:
> trainer.train()
> \`\`\`
> 
> \### Issue Severity
> 
> High: It blocks me from completing my task.

---

_[View the full topic](https://discuss.ray.io/t/rllib-gpu-memory-leak-tune-ppo-policy-server-client/10763)._
