This may be a very dumb question, but is there a way to store the best model of an experiment without saving it to disk with checkpoints? I find myself having to run a hyperparameters search, but I want to avoid having to save to disk periodically. I want to keep track of the best model (or other objects) in memory (e.g. in variable) instead.
I tried doing it by simply storing it in a static variable from the train function, but this doesn’t seem to work, the training function when run within a ray optimization does not communicate with the rest of the code. Yet, if I return it from the training function it will be stored in the report for every single trial, filling up my memory.
Apologies if this is trivial, I’m quite new to this library.
Thank you in advance for your help!