# Offline training using previous obs+action=reward tuples

**URL:** <https://discuss.ray.io/t/offline-training-using-previous-obs-action-reward-tuples/2249>\
**Category:** RLlib\
**Created:** [May 21, 2021, 7:28pm UTC](https://discuss.ray.io/t/offline-training-using-previous-obs-action-reward-tuples/2249 "2021-05-21T19:28:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Denys\_Ashikhin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denys_ashikhin/32/58_2.png) [@Denys\_Ashikhin](https://discuss.ray.io/u/Denys_Ashikhin)\
**Post date:** [May 21, 2021, 7:28pm UTC](https://discuss.ray.io/t/offline-training-using-previous-obs-action-reward-tuples/2249/1 "2021-05-21T19:28:46Z")

</div>

Hi,

Is it possible to store a tuple of (obs, action, reward) to then use for training models? This is mainly in the case of changing hyperparameters and instead of rerunning expensive models/environments we can use previous data to speed up training to a degree.

Thanks in advance,  
Denys A.

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [May 24, 2021, 10:13am UTC](https://discuss.ray.io/t/offline-training-using-previous-obs-action-reward-tuples/2249/2 "2021-05-24T10:13:15Z")

</div>

Hey @Denys_Ashikhin , yes, this is usually don by our off-policy algorithms, like DQN, SAC, DDPG, CQL, and TD3.  
If you look at their execution plans (e.g. `ray/rllib/agents/dqn/dqn.py::execution_plan`), you will see that we create a `LocalReplayBuffer` in there that’s used for storing experience tuples from the environment rollouts and re-use the samples therein repeatedly for the training updates.