# Accessing rllib evaluation in tune.Analysis
**URL:** https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490
**Category:** Ray Tune
**Created:** [March 31, 2021, 2:56pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490 "2021-03-31T14:56:04Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@MaximeBouton](https://discuss.ray.io/u/MaximeBouton)
#### Post date: [March 31, 2021, 2:56pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490/1 "2021-03-31T14:56:04Z")
When running multiple RL experiments with evaluation during training, rllib reports the evaluation metrics to tensorboard as “ray/tune/evaluation/…”.
How is it possible to access those metrics programmatically for _analysis after training_?
I looked into `tune.Analysis` to easily get statistics about multiple experiments, it works great but it has everything except the evaluation data 😕
```python
import ray.tune as tune
EXPERIMENT_FOLDER = "/home/username/ray_results/my_experiments"
analysis = tune.Analysis(EXPERIMENT_FOLDER, default_metric="episode_reward_mean", default_mode="max")
df = analysis.dataframe()
for c in df.columns: print(c)
```
This will print:
```auto
episode_reward_max
episode_reward_min
episode_reward_mean
episode_len_mean
episodes_this_iter
...
custom_metrics/...
...
config/...
```
I am looking for a similar solution to get evaluation data. Thanks
Using Ray version 1.2.0.
---
### Author:  [@sven1977](https://discuss.ray.io/u/sven1977)
#### Post date: [March 31, 2021, 3:10pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490/2 "2021-03-31T15:10:11Z")
@MaximeBouton actually, I’m not sure. Could you ask this under Ray Tune?
The evaluation data gets returned under the “evaluation” top-level key within the metrics dict that an RLlib Trainer.train() returns.
@kai @amogkam @rliaw ?
---
### Author:  [@MaximeBouton](https://discuss.ray.io/u/MaximeBouton)
#### Post date: [March 31, 2021, 4:05pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490/3 "2021-03-31T16:05:46Z")
Thank you for the answer.
From my own investigation, I found that `tune` is using `result.json` to pull the data, and the “evaluation” key is not reported in this file.
(I changed the topic to ray tune)
---
### Author:  [@MaximeBouton](https://discuss.ray.io/u/MaximeBouton)
#### Post date: [March 31, 2021, 4:21pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490/4 "2021-03-31T16:21:07Z")
Could this has something to do with the fact that initially the dict returned by `Trainer.train()` does not have the key? If evaluation is done every n iterations?
---
### Author:  [@LukasNothhelfer](https://discuss.ray.io/u/LukasNothhelfer)
#### Post date: [June 17, 2021, 3:00pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490/5 "2021-06-17T15:00:41Z")
I would also be very interested in a solution to this. Does the problem still exist?
---
### Author:  [@mannyv](https://discuss.ray.io/u/mannyv)
#### Post date: [June 17, 2021, 8:47pm UTC](https://discuss.ray.io/t/accessing-rllib-evaluation-in-tune-analysis/1490/6 "2021-06-17T20:47:30Z")
@MaximeBouton @LukasNothhelfer
Take a look at this message and the one below it for an explanation on the issue and how to fix it.
> [@Saving checkpoints with good custom\_metric using tune.run()](https://discuss.ray.io/t/saving-checkpoints-with-good-custom-metric-using-tune-run/2109/11):
>
> Glad it worked. I figured out the issue with the csv logger. The very first time it logs data using the “on\_result” method is when it creates the file and determines the “fieldnames” (flattened keys) that it will log for the duration of the experiment. In an example like yours where you are doing evaluations n \> 1 the evaluation keys will not be in that first set of results and so in subsequent calls to “on\_result” they will be ignored. This will also be true for your custom\_metrics keys if yo…
> [@Saving checkpoints with good custom\_metric using tune.run()](https://discuss.ray.io/t/saving-checkpoints-with-good-custom-metric-using-tune-run/2109/12):
>
> At least for this example here using the json log does work with these changes applied. --- orig/ray/tune/analysis/experiment\_analysis.py 2021-05-12 22:18:02.195126415 -0400 +++ fixed/ray/tune/analysis/experiment\_analysis.py 2021-05-12 22:15:36.774961258 -0400 @@ -17,7 +17,7 @@ from ray.tune.error import TuneError from ray.tune.result import DEFAULT\_METRIC, EXPR\_PROGRESS\_FILE, \ - EXPR\_PARAM\_FILE, CONFIG\_PREFIX, TRAINING\_ITERATION + EXPR\_PARAM\_FILE, CONFIG\_PREFIX, TRAINING\_ITERATION, …