# Cannot reproduce training results in evaluation even on same dataset

**URL:** <https://discuss.ray.io/t/cannot-reproduce-training-results-in-evaluation-even-on-same-dataset/8280>\
**Category:** RLlib\
**Created:** [November 15, 2022, 12:11pm UTC](https://discuss.ray.io/t/cannot-reproduce-training-results-in-evaluation-even-on-same-dataset/8280 "2022-11-15T12:11:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![SVH](https://avatars.discourse-cdn.com/v4/letter/s/85f322/32.png) [@SVH](https://discuss.ray.io/u/SVH)\
**Post date:** [November 15, 2022, 12:11pm UTC](https://discuss.ray.io/t/cannot-reproduce-training-results-in-evaluation-even-on-same-dataset/8280/1 "2022-11-15T12:11:29Z")

</div>

- High: It blocks me to complete my task.

I have a model that seems to train very well. Mean reward converges well above 100 including variance. When I try to serve the model or evaluate it running on the SAME dataset it only produces a mean reward below 40. Im struggling to figure out how to have the trained model produce similar performance as suggested by training.

About the model:

1. Its a custom env that on each reset randomly resets the state to one of 328 samples
2. The model will always only return done when there are no more observations in the sample which is around 100-140 steps
3. There can be small rewards before episode termination but most positive / negative rewards will be at the end of the sample

Training

```auto
env_name = "my_env"
register_env(env_name, env_creator)

experiment = tune.run(
    "PPO",
    config={
        "env": env_name,
            #"framework": "tf2",
            #"eager_tracing":True,
            #"lambda": 0.95,
            #"kl_coeff": 0.5,
            #"clip_rewards": True,
            #"clip_param": 0.3,
            #"vf_clip_param": 10.0,
            #"vf_share_layers": True,
            #"vf_loss_coeff": 1e-2,
            #"entropy_coeff": 0.01,
            #"train_batch_size": 10000,
            #"rollout_fragment_length": 140,
            #"sample_batch_size": 130,
            #"sgd_minibatch_size": 130,
            #"num_sgd_iter": 10,
            "num_workers": 6,
            #"num_envs_per_worker": 16,
            #"lr": 0.0001,
            "gamma": 1.0,
            "batch_mode": "complete_episodes",
            "metrics_smoothing_episodes": 300,
            #"num_cpus": 4
         
    },
    metric="episode_reward_mean",
    mode="max",
    stop={"training_iteration": 250},
    checkpoint_at_end=True,
)

```

Evaluation

```auto
register_env(env_name, env_creator)

config = ppo.PPOConfig()
config.explore=False
agent = config.build(env=env_name)
agent.restore(checkpoint_path)

env = env_creator(config)
state = env.reset()

sum_reward = 0

episodes = 1
while True:
    #action = agent.compute_single_action(state)
    action = agent.compute_action(state)
    state, reward, done, info = env.step(action)

    #if(reward != 0):
    # print(reward)
    sum_reward += reward
    
    if done:
        if (episodes == 328):
            break
        else:
            state = env.reset()
            episodes += 1;

print(sum_reward)
print(episodes)
print(sum_reward / episodes)

```

MEan reaward accross episodes are closer to 40 than to 100+.

Im struggling to figure out why I cannot produce the training results even on the same dataset used for training. I did try to increate batch\_size to 10.000 to have around 100 out of 328 samples fully play out in training.

![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/5/56484b7e1c3305cc3f1dbca66ba71c5c671d1302.png)

If possible does anyone have some hands on thing I can try?

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [November 20, 2022, 12:53pm UTC](https://discuss.ray.io/t/cannot-reproduce-training-results-in-evaluation-even-on-same-dataset/8280/2 "2022-11-20T12:53:50Z")

</div>

@SVH This has been answered by @mannyv in the [reply](https://discuss.ray.io/t/training-mean-reward-vs-evaluation-mean-rewward/8273/2) to a similar problem of yours, I guess.

The `explore` attribute for evaluation has to be set to `True` to achieve comparable results to training.