ExperimentAnalysis - Did you pass the correct `metric` parameter?

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Questions
I run experiments and reported metrics and checkpoints by calling:

            session.report(
                {"mrr": mrr, "mr": mr, "hits10": hits10, "hits5": hits5, "hits3": hits3, "hits1": hits1},
                checkpoint=checkpoint)

Then I create an ExperimentAnalysis object:

analysis = ExperimentAnalysis(experiment_checkpoint_path=RUN_NAME)

Then I call the method get_best_logdir as follows:

best_run_name = analysis.get_best_logdir("mrr", mode="max")

and get the error message Could not find best trial. Did you pass the correct metric parameter?. However, if I restore the experiment with Tuner.restore, can access the mrr metric.

Why is the ExperimentAnalysis not working as expected?

2

Hi @mblum,

Could you try using the new ResultGrid API instead? See here for a guide: Analyzing Tune Experiment Results — Ray 2.3.0

result_grid = Tuner.restore("/path/to/experiment").get_results()
best_run_path = result_grid.get_best_result(metric="mrr", mode="max").uri

To figure out the existing issue, what is the RUN_NAME that you’re passing into ExperimentAnalysis?

3

Hi @justinvyu,

thank you very much for your help. I have tried out your code and got the same error:
RuntimeError: No best trial found for the given metric: mrr. This means that no trial has reported this metric, or all values reported for this metric are NaN. To not ignore NaN values, you can set the `filter_nan_and_inf` arg to False.

The RUN_NAME is the path to the Ray experiment (‘’/media/compute/homes/mblum/ray_results/feature_vs_objective_link_pred_2023-02-23_10-36-24"). However, the path must be correct because the restoring seems to work correctly. Moreover, if I call results._experiment_analysis._trial_dataframes, the dataframes contain the mrr score.

[1 rows x 19 columns], '/media/compute/homes/mblum/ray_results/feature_vs_objective_link_pred_2023-02-23_10-36-24/train_89bf9_00003_3_alpha=0.5905,batch_norm=False,batch_size=256,dim=100,dropout=0,eta=1,lr=0.0002,reg=False_2023-02-23_10-36-46':               
mrr  time_this_iter_s  should_checkpoint  done  timesteps_total  episodes_total  training_iteration     trial_id                     experiment_id  ...   timestamp  time_total_s   pid  hostname         node_ip time_since_restore  timesteps_since_restore  iterations_since_restore  warmup_time
0  tensor(0.0018)        288.402447               True  True              NaN             NaN                   1  89bf9_00003  7e67e64d429e43038068b31bdee51141  ...  1677145300    288.402447  8790     handy  129.70.137.103         288.402447                        0                         1     0.004892
4

@mblum The issue seems to be that your metric is passed to Tune as a tensor object, rather than a float. Could you try reporting mrr.item() (if you’re using PyTorch) instead?

5

@justinvyu That’s what I am trying out right now. It seems to work. However, that’s bad news for the experiments I have to re-run.

6

Got it, you could also find the best result through the dataframe directly if you don’t want to re-run the experiments. Generally, you should report primitive types through session.report(metrics), but it technically does allow all sorts of serializable objects to be reported.

So, it’s hard to throw a warning during logging. We may want to add a try/catch to convert to a primitive type, when using it to order trials.

7

Thank you very much @justinvyu ! I will follow your suggestion.

You are right, some warning or automatic type conversion would be great. However, making this more clear in the documentation would actually help a lot in the future.