Hello! I’ve recently started using Ray.io and I think it’s awesome. However I’m struggling to load my results after the tuning ends. I tried both Tuner.restore and ExperimentAnalysis(path) with the same results.
Problem:
I’m running my hyper parameter tuning on a cluster. After it’s done I’d like to have access to the trial data to visualise them (with pareto front). How can I do that?
When I run it locally and load directly from the ~/ray_results/experiment directory, it works. However I noticed that when I change the experiment directory name I have an error:
Could not fetch metrics for xxx: both result.json and progress.csv were not found
And I can see in the error that ray tries to load them from the old, non existent directory.
It seems that there are some hardcoded absolute paths in the experiment state?
I think this is the reason why I cannot load my results from the cluster as well - I download them using ray rsync-down, meaning some paths in the experiment don’t work anymore.
How this should be done? How can I access my results in the cluster locally to analyse them?
EDIT:
Yeah I can see in the error paths that they refer to home/ray/ray_results(...), path from my cluster.
Is there a way to update that when loading results into the Tuner?
I’m using ray 2.7.1