Best way to pass large objects from Trainable?

In order to save time later on, I would like to pass fitted estimators and their CV predictions from outside the Trainable. What would be the best way to do so? I would not want to save them to disk as they would be loaded right away, and tune.report, while convenient, sounds to me like it would not work well in this instance (though I of course may be wrong).

Sorry for the slow reply!

tune.report could work. This is an interesting use case :slight_smile: I would suggest doing something like:

id = ray.put(obj)
tune.report(array=id)

and then fetching it from the trial.last_result upon completion.

Let me know if that works for you!

Thanks, I’ll try it out!

Just wanted to quickly emphasize that object IDs should be used here. Tune results will be logged to CSV and JSON and putting large objects here (e.g. datasets) would blow up the logfiles and bottleneck the serialization process.

Thanks for letting me know, Kai - this is exactly the kind of stuff I was worried about.

I’ve had issues with passing the IDs and then getting them as the objects were deleted, but I managed to take care of it by using a separate named actor. It all works well now. Thanks!

1 Like