Getting analysis results from Tune before it's finished

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I’m using Tune to run my experiments under different configurations. My code looks like this:

analysis = ray.tune.run(
    ray.Tune.Experiment(...)
)

analysis.dataframe().to_csv('result.csv')

This works and I get a CSV with all the results. However, I have to wait until all the experiments are finished. Sometimes I start a run that takes two days. I want to start seeing some values on that CSV without having to wait for all the runs to be finished. How can I do that?

Thanks,
Ram.

2

Which version of Ray are you running?
If ExperimentAnalysis object works well with you, you can take a look at how it’s initiated. Maybe you can just construct it with ExperimentAnalysis(experiment_checkpoint_path)?

Also tensorboard should give you some visuals of trial results over time as well.

3

I hoped that this was a common enough need that other people have already figured out a solution for it…

To your questions:

I’m running Ray 2.1.0.

I could create an ExperimentAnalysis object like you suggest. I guess I’d have to create a separate thread that calls it once in a while. But I see that in the initialization of ExperimentAnalysis one of the arguments is trials. How can I get these for an experiment that ray.tune.run hasn’t finished and returned yet?

Thanks,
Ram.

4

yeah I think you are right in that it should be a supported feature. I filed [tune] support viewing partial experiment result as tuning goes on · Issue #34207 · ray-project/ray · GitHub to track it so that the team can prioritize on it.

Given what we have now, theoretically everything is already snapshotted periodically into the experiment folder. This also includes all trials. You may load back the trials in a similar fashion like here.

5

Thank you so much for filing that feature request!

6

@cool-RR Did you get all your questions answered from @xwjiang2010? If so may I resolve this issue?

7

Yes, thank you Jules.

8

@cool-RR what is the information contained in the CSV that you are looking for? Is this information not being output in the Tune Console Output?

9

@matthewdeng All of the metrics recorded by the experiment. Example CSV file. I don’t think they all show up in the console output, and even if they did, this would not be a convenient way to consume them.