# Training steps for DQN

**URL:** <https://discuss.ray.io/t/training-steps-for-dqn/14442>\
**Category:** Configure Algorithm, Training, Evaluation, Scaling\
**Created:** [April 18, 2024, 12:51pm UTC](https://discuss.ray.io/t/training-steps-for-dqn/14442 "2024-04-18T12:51:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenjaminNatali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/benjaminnatali/32/6015_2.png) [@BenjaminNatali](https://discuss.ray.io/u/BenjaminNatali)\
**Post date:** [April 18, 2024, 12:51pm UTC](https://discuss.ray.io/t/training-steps-for-dqn/14442/1 "2024-04-18T12:51:23Z")

</div>

How do I set up the training configuration of a DQN algorithm in order to be able to control the number of training steps performed in one iteration of algo.train()?

---

<div class="post-metadata">

**Author:** ![VisionZUS29](https://avatars.discourse-cdn.com/v4/letter/v/ea666f/32.png) [@VisionZUS29](https://discuss.ray.io/u/VisionZUS29)\
**Post date:** [April 18, 2024, 3:12pm UTC](https://discuss.ray.io/t/training-steps-for-dqn/14442/2 "2024-04-18T15:12:43Z")

</div>

As in, the number of steps per episode before your environment gets reset and another episode is performed?

---

<div class="post-metadata">

**Author:** ![BenjaminNatali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/benjaminnatali/32/6015_2.png) [@BenjaminNatali](https://discuss.ray.io/u/BenjaminNatali)\
**Post date:** [April 18, 2024, 3:29pm UTC](https://discuss.ray.io/t/training-steps-for-dqn/14442/3 "2024-04-18T15:29:19Z")

</div>

No, as in the number of steps corresponding to one training iteration, I.e. to one call to algo.train()

---

<div class="post-metadata">

**Author:** ![VisionZUS29](https://avatars.discourse-cdn.com/v4/letter/v/ea666f/32.png) [@VisionZUS29](https://discuss.ray.io/u/VisionZUS29)\
**Post date:** [April 18, 2024, 4:01pm UTC](https://discuss.ray.io/t/training-steps-for-dqn/14442/4 "2024-04-18T16:01:51Z")

</div>

You could try something like this

```auto
custom_checkpoint_dir = "./your_directory"

print_guide = ["episodes_this_iter", "episode_reward_mean", "episode_reward_max", "episode_reward_min",
               "policy_reward_mean", "done",
               "num_env_steps_sampled_this_iter", ]
algo = config.build()
totalenvsteps=0

print("BEGINNING TRAINING")
for i in range(1, 1000):
    result = algo.train()
    print(f"Iteration: {i}")
    for key in print_guide:
        print(f'{key}: {result[key]}')
    if i % 20 == 0:
        checkpoint_dir = algo.save(custom_checkpoint_dir).checkpoint.path
        print(f"Checkpoint saved in directory {checkpoint_dir}")
    totalenvsteps+=result["num_env_steps_sampled_this_iter"]
    print("Total Training Steps:", totalenvsteps)
    print("\n")

```

or in your case

```auto
totalenvsteps=0

print("BEGINNING TRAINING")
while True:
    result = algo.train()
    for key in print_guide:
        print(result[key])
        print(result)
    totalenvsteps+=result["num_env_steps_sampled_this_iter"]
    print("Total Training Steps:", totalenvsteps)
    if totalenvsteps>=4000:
        break

```

**because I dont believe you could specify the exact number through the config if your env varies in number of timesteps per episode This link [here](https://github.com/ray-project/ray/issues/7466) is pretty much the question you are asking)**

now if you use ray tune you could specify how many timesteps the training runs by something like this

```auto
stop = {
        "episode_reward_mean": args.stop_reward,
        "timesteps_total": args.stop_timesteps,
    }
    # Get the path to the desktop
    desktop_path = os.path.join(os.path.join(os.environ['USERPROFILE']), 'Desktop')

    # Create a path to the RayResults folder on the desktop
    ray_results_path = os.path.join(desktop_path, 'RayResults')
    results = tune.Tuner(PPO, run_config=air.RunConfig(storage_path=ray_results_path, name=f"Ray_{date.today()}",stop=stop, verbose=1), param_space=config,).fit()

    if args.run_as_test:
        check_learning_achieved(results, args.stop_reward)

    ray.shutdown()

```

where the you can define total timesteps in the args