Add trials to experiment for later analysis

Ray Libraries (Data, Train, Tune, Serve)
1

Hi!
I am running an experiment with

tuner = tune.Tuner(trainable_with_resources, param_space=config, tune_config=tune_config, run_config=run_config)
results = tuner.fit()

That’s all nice, but what if I want to add trials to my experiment later, such that I can do something like

analysis = ExperimentAnalysis((run_config.storage_path)

for all trials within run_config.storage_path?

  • If I just start a new experiment and save it into run_config.strorage_path, only the new trials will be read for analysis.
  • Tuner.restore is only meant for incomplete experiments

I have not found a solution that works for me, yet…Thank you!

2

Hi @Jaydeen , do you mean you want to save two experiment results into the same storage path? May I know your use case?

Currently ExperimentAnalysis doesn’t support it. For result analysis, maybe you can merge multiple result.json files in different experiment folders.

3

Hi! Thanks for your reply! Say I conducted a grid_search for parameters A and B with the param_space:

A: tune.grid_search([1,2,3])
B: tune.grid_search([4,5])

At a later time I want to add the option B = 6 to my search space. How would I do that? Yes, I can run a new experiment with

A: tune.grid_search([1,2,3])
B: tune.grid_search([6])

but if I want to analyse my results later I would somehow need to merge the results manually. It would be nice if I could just add the runs to the original experiment folder …

Hope that was clearer? Thank you!

4

Got it. Thank you @Jaydeen. Currently Ray Tune doesn’t support this feature yet, you have to manually merge the result files and do some post-process.
I think this is a reasonable use case, we’ll discuss internally whether to support this feature.