Accessing rllib evaluation in tune.Analysis

[mannyv]

@MaximeBouton @LukasNothhelfer

Take a look at this message and the one below it for an explanation on the issue and how to fix it.

Saving checkpoints with good custom_metric using tune.run()

Glad it worked. I figured out the issue with the csv logger. The very first time it logs data using the “on_result” method is when it creates the file and determines the “fieldnames” (flattened keys) that it will log for the duration of the experiment. In an example like yours where you are doing evaluations n > 1 the evaluation keys will not be in that first set of results and so in subsequent calls to “on_result” they will be ignored. This will also be true for your custom_metrics keys if yo…

Saving checkpoints with good custom_metric using tune.run()

At least for this example here using the json log does work with these changes applied. --- orig/ray/tune/analysis/experiment_analysis.py 2021-05-12 22:18:02.195126415 -0400 +++ fixed/ray/tune/analysis/experiment_analysis.py 2021-05-12 22:15:36.774961258 -0400 @@ -17,7 +17,7 @@ from ray.tune.error import TuneError from ray.tune.result import DEFAULT_METRIC, EXPR_PROGRESS_FILE, \ - EXPR_PARAM_FILE, CONFIG_PREFIX, TRAINING_ITERATION + EXPR_PARAM_FILE, CONFIG_PREFIX, TRAINING_ITERATION, …