# Load prior \`tune.run()\` results from disk

**URL:** <https://discuss.ray.io/t/load-prior-tune-run-results-from-disk/4495>\
**Category:** Ray Tune\
**Created:** [December 20, 2021, 9:58pm UTC](https://discuss.ray.io/t/load-prior-tune-run-results-from-disk/4495 "2021-12-20T21:58:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![addisonklinke](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/addisonklinke/32/1906_2.png) [@addisonklinke](https://discuss.ray.io/u/addisonklinke)\
**Post date:** [December 20, 2021, 9:58pm UTC](https://discuss.ray.io/t/load-prior-tune-run-results-from-disk/4495/1 "2021-12-20T21:58:10Z")

</div>

Based Ray’s [docs](https://docs.ray.io/en/master/tune/api_docs/analysis.html?highlight=load%20results%20from%20disk#analysis-tune-analysis), I would expect the following to load the `analysis` object from a previous `tune.run()` process

```python
from ray.tune import ExperimentAnalysis
analysis = ExperimentAnalysis("~/ray_results/example-experiment")

```

I used the RayTune + PyTorch Lightning [tutorial](https://docs.ray.io/en/master/tune/tutorials/tune-pytorch-lightning.html) as a guide for working with my own model/dataset. When the process completed, I saw the `analysis.best_config` printed to console, so I believe everything ran correctly. However, if I run the above snippet to try and load the analysis back into memory, I get the following error message

```nohighlight
Could not load trials from experiment checkpoint. 
This means your experiment checkpoint is likely faulty or incomplete, 
and you won't have access to all analysis methods. 
Observed error: No module named ...

```

At first the module was `gym`, and then after pip installing now it’s `tree`. My model doesn’t use either of these, so I’m confused where these requirements are coming from. Before launching a large search space on a remote machine, I want to make sure that I can reliably retrieve the results of a prior `tune.run()` from disk in case the process crashes before printing `analysis.best_config` (in which case all the time spent on trials would’ve been wasted)

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [December 21, 2021, 8:37am UTC](https://discuss.ray.io/t/load-prior-tune-run-results-from-disk/4495/2 "2021-12-21T08:37:33Z")

</div>

Hey @addisonklinke, thanks for reporting this. I took a look and found this workflow to be a little confusing/misleading as well. I created a Github issue to track this here: [[Bug][Tune] Loading `ExperimentAnalysis` requires `Trainable` to be registered. · Issue #21212 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/21212).

In short, the primary issue seems to be that the Trainable is not registered beforehand. As a result, the code tries to import some pre-defined RLlib Trainables, which then causes the error message you’ve observed.

To resolve this, you can add the following lines of code:

```python
from ray.tune import register_trainable
register_trainable(training_function_name, training_function)

```

`training_function_name` should be the name of your `training_function`, but you can verify what the code expects this name to be by searching for `trainable_name` in the `"~/ray_results/example-experiment/experiment_state-<date>.json` file.

---

<div class="post-metadata">

**Author:** ![addisonklinke](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/addisonklinke/32/1906_2.png) [@addisonklinke](https://discuss.ray.io/u/addisonklinke)\
**Post date:** [December 21, 2021, 3:30pm UTC](https://discuss.ray.io/t/load-prior-tune-run-results-from-disk/4495/3 "2021-12-21T15:30:23Z")

</div>

@matthewdeng Thanks for the tip. Follow the PyTorch Lightning tutorial, would this be the correct implementation of your suggestion?

```python
register_trainable('train_mnist_tune', train_mnist_tune)

```

For my training function (and I suspect the MNIST example as well), this raises

```nohighlight
ValueError: Unknown argument found in the Trainable function. 
The function args must include a 'config' positional parameter. 
Any other args must be 'checkpoint_dir'

```

How then can I have a training function which takes other parameters such as number of GPUs, path to data on disk, etc? Do all of these belong in the config?

**EDIT**

By adding `checkpoint_dir=None` as the second parameter to `train_mnist_tune()`, I am able to avoid the above value error and include additional parameters. However, once `tune.run()` completes, I still get the original “could not load trials from experiment checkpoint” error when trying to initialize an `ExperimentAnalysis` object from the `~/ray_results` folder.

I noticed the population based training [section](https://docs.ray.io/en/master/tune/tutorials/tune-pytorch-lightning.html#adding-checkpoints-to-the-pytorch-lightning-module) of the Lightning tutorial mentions adding checkpoints via `TuneReportCheckpointCallback`, so I tried this. Now there are checkpoint files in each trial folder, but it did not fix the analysis loading

**EDIT 2**

I see now that `register_trainable()` must be called within the _same_ process that initializes `ExperimentAnalysis`. Previously I thought the registration was supposed to occur in the original `tune.run()` process, and that’s why it was not fixing the issue. With this modification, the proposed workaround is successful

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [December 21, 2021, 5:53pm UTC](https://discuss.ray.io/t/load-prior-tune-run-results-from-disk/4495/4 "2021-12-21T17:53:57Z")

</div>

Awesome! I’m glad you were able to find success here, and thanks for sharing the steps you took (if anyone else runs into this same issue I think they’ll really appreciate this).