# Algorithm.train does not terminate for custom env

**URL:** <https://discuss.ray.io/t/algorithm-train-does-not-terminate-for-custom-env/9978>\
**Category:** Configure Algorithm, Training, Evaluation, Scaling\
**Created:** [March 30, 2023, 5:31pm UTC](https://discuss.ray.io/t/algorithm-train-does-not-terminate-for-custom-env/9978 "2023-03-30T17:31:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![timgrewe](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/timgrewe/32/4089_2.png) [@timgrewe](https://discuss.ray.io/u/timgrewe)\
**Post date:** [March 30, 2023, 5:31pm UTC](https://discuss.ray.io/t/algorithm-train-does-not-terminate-for-custom-env/9978/1 "2023-03-30T17:31:47Z")

</div>

I use an environment generated by [NS3-Gym](https://github.com/tkn-tub/ns3-gym) and am experiencing an issue where RLlib does not terminate after the number of steps I set it to.  
For testing purposes I set `rollout_fragment_length`, `train_batch_size`, and `horizon` all to 1 and `batch_mode` to `complete_episodes` (although, from my understanding, this shouldn’t make a difference if `horizon` is 1).

I am now expecting the agent to execute 1 step (= 1 episode) and then terminate. When using one of the default environments (I used `CartPole-v1` for testing) this seems to work fine as the whole script finishes in about 20 s. However, when I’m using my own environment, the env just gets reset after 1 step and `train()` continues to run. I know that RLlib recognizes that a step has passed, since a) it executed the step (duh) and b) the environment normally would not reset after 1 step. I’m honestly at a loss as to why it just won’t stop training so any help would be greatly appreciated.

Attached below is the my training script. Since the env is bound to an external NS3 simulation, this won’t run on its own but I figured it still can’t hurt to attach it.

```auto
import ray
import ray.rllib.algorithms.dqn as dqn
from ray.tune.registry import register_env
import gym
from ns3gym import ns3env

def env_creator (env_config) -> gym.Env:
    env = ns3env.Ns3Env(port=0, startSim=True)
    return env

register_env("my_env", env_creator)

ray.init()
config = dqn.DEFAULT_CONFIG.copy()
config["num_gpus"]=0
config["num_workers"]=1
config["framework"]="tf2"
config["eager_tracing"]=True
config["disable_env_checking"] = True # Reset time is quite high so I disabled env checking
config["rollout_fragment_length"] = 1
config["batch_mode"] = "complete_episodes"
config["train_batch_size"] = 1
config["horizon"] = 1

algo = dqn.DQN(config=config, env="my_env")
algo.train()

```

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [May 1, 2023, 8:49pm UTC](https://discuss.ray.io/t/algorithm-train-does-not-terminate-for-custom-env/9978/2 "2023-05-01T20:49:38Z")

</div>

I’m not sure if I understand.  
Your goal is to reset the env after one step.  
So as per the following sentence, you’ve achieved that already:

> However, when I’m using my own environment, the env just gets reset after 1 step and train() continues to run.

But now `train()` continues to run? What’s the issue with that? That is intended behaviour as trainings generally continue for ever if you don’t introduce stopping conditions.

---

<div class="post-metadata">

**Author:** ![timgrewe](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/timgrewe/32/4089_2.png) [@timgrewe](https://discuss.ray.io/u/timgrewe)\
**Post date:** [May 10, 2023, 2:50pm UTC](https://discuss.ray.io/t/algorithm-train-does-not-terminate-for-custom-env/9978/3 "2023-05-10T14:50:40Z")

</div>

Maybe I phrased my question wrong but doesn’t `train` usually only execute one episode?  
That’s what I was experiencing with the `CartPole` env at least.  
Doesn’t really matter since I’ve gotten this to work by switching from the conf dictionary to using `DQNConfig`. Cheers for the reply anyway!