# How to train until convergence

**URL:** <https://discuss.ray.io/t/how-to-train-until-convergence/6738>\
**Category:** RLlib\
**Created:** [July 6, 2022, 10:06am UTC](https://discuss.ray.io/t/how-to-train-until-convergence/6738 "2022-07-06T10:06:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![carlorop](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@carlorop](https://discuss.ray.io/u/carlorop)\
**Post date:** [July 6, 2022, 10:06am UTC](https://discuss.ray.io/t/how-to-train-until-convergence/6738/1 "2022-07-06T10:06:21Z")

</div>

Here it is an example of the configuration I am using for running some models:

```python

    experiment_params = {
            "training": {
                "env": "example",
                "run": args.algorithm,
                "stop": {"training_iteration": 2000
                         #"timesteps_total": 8000000,
                        
                        },
                "local_dir": "/opt/ml/output/intermediate",
                "checkpoint_at_end": True,
                "checkpoint_freq": 10,
                "config": {
                    "num_workers": int(os.cpu_count())-1,                    
                    #"sigma0": 0.01,
                    "lr": 0.001,
                    # "sample_batch_size": 1,
                    "num_gpus": num_gpus,
                    "gamma": float(args.gamma),
                    "seed":args.seed,
                    # RAY 0.5 syntax: gpu_fraction
                    #"prioritized_replay": False,
                },

            }
        }
    ray.tune.run_experiments(copy.deepcopy(experiment_params))

```

I am using RLLIB 0.8.5, is there any way to change the stopping criteria such that it stops training once episode reward mean has converged?. I wouldn’t mind updating RLLIB

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [July 6, 2022, 4:01pm UTC](https://discuss.ray.io/t/how-to-train-until-convergence/6738/2 "2022-07-06T16:01:40Z")

</div>

Hi! Definition of convergence depends on the problem and algorithm. Tune can only stop on metrics that are reported to it. So you’ll have to implement a metric that digests a record of past mean rewards to provide a measurement of convergence for tune to observe.  
Here’s what to do:

1. Write a custom Callback that tracks your metric of interest over time  
1.1 Write metric (maybe a simple average of absolute change in episode reward mean) into  
results
2. Tell tune to monitor that metric as a stopping metric

See [here](https://github.com/ray-project/ray/blob/master/rllib/examples/custom_metrics_and_callbacks.py) for example.