# DD-PPO RolloutWorker Hangs

**URL:** <https://discuss.ray.io/t/dd-ppo-rolloutworker-hangs/1730>\
**Category:** RLlib\
**Created:** [April 14, 2021, 9:35pm UTC](https://discuss.ray.io/t/dd-ppo-rolloutworker-hangs/1730 "2021-04-14T21:35:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Zhang](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@Mark\_Zhang](https://discuss.ray.io/u/Mark_Zhang)\
**Post date:** [April 14, 2021, 9:35pm UTC](https://discuss.ray.io/t/dd-ppo-rolloutworker-hangs/1730/1 "2021-04-14T21:35:34Z")

</div>

Hi guys,

I am using DD-PPO for a GPU-required environment.

For testing purpose, I started the learning with only four rollout workers. It can learn for a few iterations, but then the learning stops silently. When I check the GPU usage after the learning stops, I can see the following:

```
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 1 19824 C ray::RolloutWorker.par_iter_next() 1875MiB |
| 1 19822 C ray::RolloutWorker 1875MiB |
| 1 19823 C ray::RolloutWorker 1877MiB |
| 1 19821 C ray::RolloutWorker 1877MiB |
+-----------------------------------------------------------------------------+

```

This lasts forever.

It looks to me that the worker with PID 19824 hangs in the process of `par_iter_next()` while all other workers are waiting. The time this problem occurs is random: somethings in the first iteration, sometimes later.

Do you have any comments on the possible cause for this? Any suggestion on how to debug (e.g., how to check at which step the rollout worker hangs)? Thanks a lot!

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [April 20, 2021, 4:41pm UTC](https://discuss.ray.io/t/dd-ppo-rolloutworker-hangs/1730/2 "2021-04-20T16:41:26Z")

</div>

Hey @Mark_Zhang , sorry, I’m not sure what this could be. Could you file a github issue and assign it to me with a self-sufficient reproduction script?

Thanks

---

<div class="post-metadata">

**Author:** ![Mark\_Zhang](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@Mark\_Zhang](https://discuss.ray.io/u/Mark_Zhang)\
**Post date:** [May 6, 2021, 11:11pm UTC](https://discuss.ray.io/t/dd-ppo-rolloutworker-hangs/1730/3 "2021-05-06T23:11:29Z")

</div>

Thanks @sven1977, I figured it out while I was working on the self-sufficient example.

Sorry I missed a detail in the earlier post as I thought it was not related, but it does.

I was trying to make DD-PPO work with multi-agent environment (multi-agent env with agents could have early dones; agents also share the same policy).

I removed the assertion at [this line](https://github.com/ray-project/ray/blob/master/rllib/agents/ppo/ddppo.py#L173) since it does not stand for multi-agent environment training. I thought it shouldn’t matter as all sampled batch experience are used for training the same policy. But this causes the hanging problem if the batches from different workers (with different sizes) are divided into different numbers of mini-batches using the same config[“sgd\_minibatch\_size”]. Please see illustration below. This also explains the random occurrence of the hanging.

My workaround for this is to use an adaptive mini-batch size, i.e., “batch.size/NUM\_MINI\_BATCH”, to make sure every worker gets the same mini-batch number, and this worked out. Any comments on this? More graceful ways to handle this?

 ![Screen Shot 2021-05-06 at 4.46.20 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/9eb9653371c1611b7217f60c3d1cdd306039e8a2.png)

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [May 27, 2021, 6:50am UTC](https://discuss.ray.io/t/dd-ppo-rolloutworker-hangs/1730/4 "2021-05-27T06:50:10Z")

</div>

This looks great @Mark_Zhang ! Could you provide a PR with your fix? I’m assuming it only affects DD-PPO and if you know that it’s learning in your case, we should merge this to help others use DD-PPO in a multi-agent setting.