Say I have a custom scheduler for Ray Tune that decides to pause a trial in on_trial_result and restore it some time later. The question is, does Ray Tune have to restore it on the same node in the cluster? or, say there is a node without job to do, is it possible that Ray restores the trial on that node?
It can restore on any node. Be sure to either use a NFS or only save things in the specific trial directory.