# DQN (and maybe other algo) should take into account the "num\_envs\_per\_worker" config when computing the round robin native\_ratio used to determined the number of steps to use for training

**URL:** <https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763>\
**Category:** RLlib\
**Created:** [April 17, 2021, 11:22am UTC](https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763 "2021-04-17T11:22:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maxime\_Riche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/maxime_riche/32/731_2.png) [@Maxime\_Riche](https://discuss.ray.io/u/Maxime_Riche)\
**Post date:** [April 17, 2021, 11:22am UTC](https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763/1 "2021-04-17T11:22:45Z")

</div>

A possible improvement:

DQN (and maybe other algo) should take into account the “num\_envs\_per\_worker” config when computing the round robin native\_ratio used to determined the number of steps to use for training

When computing the native ratio below (from DQNTrainer) (to know for how many steps to train compared to the steps added in the replay buffer), we should use:

```
native_ratio = (
    config["train_batch_size"] / (config["rollout_fragment_length"]*config["num_envs_per_worker"]))

```

instead of

```
native_ratio = (
    config["train_batch_size"] / config["rollout_fragment_length"])

```

This is because the effective “rollout\_fragment\_length” is (“rollout\_fragment\_length” \* “num\_envs\_per\_worker”) as described in the doc:

```
# Divide episodes into fragments of this many steps each during rollouts.
# Sample batches of this size are collected from rollout workers and
# combined into a larger batch of `train_batch_size` for learning.
#
# For example, given rollout_fragment_length=100 and train_batch_size=1000:
# 1. RLlib collects 10 fragments of 100 steps each from rollout workers.
# 2. These fragments are concatenated and we perform an epoch of SGD.
#
# When using multiple envs per worker, the fragment size is multiplied by
# `num_envs_per_worker`. This is since we are collecting steps from
# multiple envs in parallel. For example, if num_envs_per_worker=5, then
# rollout workers will return experiences in chunks of 5*100 = 500 steps.
#
# The dataflow here can vary per algorithm. For example, PPO further
# divides the train batch into minibatches for multi-epoch SGD.
"rollout_fragment_length": 200,

```

This change is related to rllib.agents.dqn.dqn.py:

```
def calculate_rr_weights(config: TrainerConfigDict) -> List[float]:
    """Calculate the round robin weights for the rollout and train steps"""
    if not config["training_intensity"]:
        return [1, 1]
    # e.g., 32 / 4 -> native ratio of 8.0
    native_ratio = (
        config["train_batch_size"] / config["rollout_fragment_length"])
    # Training intensity is specified in terms of
    # (steps_replayed / steps_sampled), so adjust for the native ratio.
    weights = [1, config["training_intensity"] / native_ratio]
    return weights
```

---

<div class="post-metadata">

**Author:** ![Maxime\_Riche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/maxime_riche/32/731_2.png) [@Maxime\_Riche](https://discuss.ray.io/u/Maxime_Riche)\
**Post date:** [April 19, 2021, 8:09am UTC](https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763/2 "2021-04-19T08:09:04Z")

</div>

In fact we may even need to use

```
native_ratio = (
    config["train_batch_size"] / (config["rollout_fragment_length"]*config["num_envs_per_worker"]*config["num_workers"]))
```

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [April 21, 2021, 11:12am UTC](https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763/3 "2021-04-21T11:12:25Z")

</div>

Thanks @Maxime_Riche ! This looks totally reasonable. I’ll create a PR.

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [April 21, 2021, 11:20am UTC](https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763/4 "2021-04-21T11:20:01Z")

</div>

> <https://github.com/ray-project/ray/pull/15436>
>
> DQN native\_ratio (for training intensity) incorrect.
> Also see this discussion… here:
> https://discuss.ray.io/t/dqn-and-maybe-other-algo-should-take-into-account-the-num-envs-per-worker-config-when-computing-the-round-robin-native-ratio-used-to-determined-the-number-of-steps-to-use-for-training/1763/3
> 
> 
> \## Why are these changes needed?
> 
> 
> \## Related issue number
> 
> 
> \## Checks
> 
> \- \[x\] I've run \`scripts/format.sh\` to lint the changes in this PR.
> \- \[x\] I've included any doc changes needed for https://docs.ray.io/en/master/.
> \- \[x\] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
> \- Testing Strategy
> - \[x\] Unit tests
> - \[\] Release tests
> - \[\] This PR is not tested :(