Not able to resume experiment

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

I tried to use tune.Tuner.restore(path).fit() to resume an experiment, but it doesn’t work.
My code is

tuner = tune.Tuner.restore("./ray_results/PPO_2022-12-07_01-43-37")
tuner.fit()

The error is:

2022-12-07 15:04:15,887	INFO worker.py:1519 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8266 
2022-12-07 15:04:18,932	INFO experiment_analysis.py:795 -- No `self.trials`. Drawing logdirs from checkpoint file. This may result in some information that is out of sync, as checkpointing is periodic.
Traceback (most recent call last):
  File "/home/sd469/HideAndSeek/Code/venv/lib/python3.8/site-packages/ray/tune/tuner.py", line 243, in fit
    return self._local_tuner.fit()
  File "/home/sd469/HideAndSeek/Code/venv/lib/python3.8/site-packages/ray/tune/impl/tuner_internal.py", line 297, in fit
    analysis = self._fit_resume(trainable)
  File "/home/sd469/HideAndSeek/Code/venv/lib/python3.8/site-packages/ray/tune/impl/tuner_internal.py", line 431, in _fit_resume
    analysis = run(**args)
  File "/home/sd469/HideAndSeek/Code/venv/lib/python3.8/site-packages/ray/tune/tune.py", line 538, in run
    experiments[i] = Experiment(
  File "/home/sd469/HideAndSeek/Code/venv/lib/python3.8/site-packages/ray/tune/experiment/experiment.py", line 155, in __init__
    assert local_dir_path in experiment_checkpoint_dir_path.parents
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sd469/HideAndSeek/Code/venv/lib/python3.8/site-packages/ray/tune/tuner.py", line 245, in fit
    raise TuneError(
ray.tune.error.TuneError: The Ray Tune run failed. Please inspect the previous error messages for a cause. After fixing the issue, you can restart the run from scratch or continue this run. To continue this run, you can use `tuner = Tuner.restore("./ray_results/PPO_2022-12-07_01-43-37")`.
python-BaseException
2

Hey @Darrell_Dai, this is an error in the latest Ray release that should be fixed in the upcoming 2.2 version (which should be out in days). You can also use the latest nightly version that has the fix already included - Installing Ray — Ray 2.1.0

Let me know if that helps!

3

@Yard1 Thanks for reply. I found the nightly version doesn’t include RLlib. Any way I can get around this?

4

The nightly version definitely contains RLLib. Are you getting some sort of an error?

5

I tried to import rllib from ray, but it shows no module called rllib

6

I couldn’t reproduce this on colab - Google Colab