# DQN Rollout Config to fit Nature DQN

**URL:** <https://discuss.ray.io/t/dqn-rollout-config-to-fit-nature-dqn/10883>\
**Category:** Configure Algorithm, Training, Evaluation, Scaling\
**Created:** [June 2, 2023, 12:58pm UTC](https://discuss.ray.io/t/dqn-rollout-config-to-fit-nature-dqn/10883 "2023-06-02T12:58:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rajfly](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rajfly/32/3916_2.png) [@rajfly](https://discuss.ray.io/u/rajfly)\
**Post date:** [June 2, 2023, 12:58pm UTC](https://discuss.ray.io/t/dqn-rollout-config-to-fit-nature-dqn/10883/1 "2023-06-02T12:58:55Z")

</div>

Hi, how would I configure the **rollout settings** to fit the DQN in the nature paper? I am confused at the Rollout Config settings. Specifically, I want to exactly replicate what was done in the Nature DQN paper. So I have 0 num\_rollout\_workers (rollouts are done in the local worker). I would like for each iteration in the DQN to step the environment once and store this one timestep into the buffer. After which with a train\_batch\_size of 32, it should sample from the replay buffer 32 random samples and proceed on training the policy network.

In the the code below, I have set the batch\_size to 32, training\_intensity to be None, rollout settings to be rollout\_fragment\_length=1 and batch\_mode=‘truncate\_episodes’. I wonder if this is correct for what I am trying to achieve. I.e., at timestep\_t, store one transition to buffer (state\_t, action\_t, reward\_t, state\_t+1) and then randomly sample 32 transitions from the buffer to train on. At timestep\_t+1, it should store into the buffer ( (state\_t+1, action\_t+1, reward\_t+1, state\_t+2) and so on …

```auto
    param_space = DQNConfig()

    param_space = param_space.training(
        gamma=0.99, 
        lr=1e-4, 
        train_batch_size=32, 
        model={
            '_disable_preprocessor_api': True,
            'conv_filters': [[32,8,4],[64,4,2],[64,3,1]],
            'conv_activation': 'relu',
            'post_fcnet_hiddens': [512],
            'post_fcnet_activation': 'relu',
            'no_final_linear': False,
            'vf_share_layers': False,
        },
        optimizer={'adam_epsilon': 1e-8},
        grad_clip=None,
        num_atoms=1,
        noisy=False,
        dueling=False,
        double_q=False,
        n_step=1,
        replay_buffer_config={'type': 'ReplayBuffer', 'capacity': 100000},
        td_error_loss_fn='huber',
        training_intensity=None,
    )

    param_space = param_space.rollouts(num_rollout_workers=0, num_envs_per_worker=1, rollout_fragment_length=1, batch_mode='truncate_episodes')

```

---

<div class="post-metadata">

**Author:** ![direland3](https://avatars.discourse-cdn.com/v4/letter/d/65b543/32.png) [@direland3](https://discuss.ray.io/u/direland3)\
**Post date:** [June 2, 2023, 2:57pm UTC](https://discuss.ray.io/t/dqn-rollout-config-to-fit-nature-dqn/10883/2 "2023-06-02T14:57:26Z")

</div>

I think you also need to specify how many updates to do when calling `.train()`. I am also trying to figure this out: I basically just want `train()` to be a single batched update, but it doesn’t seem like this is the case and I can’t see how to change the config to make it so.