Hi, I’m trying to use Ray Tune with Pytorch lightning and I would like to be able to access the lightning module from a callback so I can use the module’s own log() method. It seems like the only way to do that is to somehow add the module to the result dict, which I’m not sure how to do.
Hey @import-antigravity , thanks for dropping by.
What are you trying to do in particular? The result dict tends to be quite light-weight.
I mean, specifically I’m trying to find a workaround because TBXLoggerCallback doesn’t work right now.
Basically, pytorch-lightning has its own methods for logging which abstract away the specifics of whether you’re using tensorboard or another logger. That being said, they’re methods of the module itself, so I want to be able to access the module from a callback in log_trial_result() somehow. Does that make sense?
Hmm, OK got it; could you try just logging as a Pytorch Lightning callback (instead of a Tune callback)?
The reason is because Tune callbacks are run on a separate process than Pytorch Lightning.
I’d run it as a lightning callback and do no logging whatsoever in Ray but I don’t think Lightning knows when Ray will terminate a trial, so there’s no way to log the hyperparameters once at the end of the trial
Hmm, could you log hyperparameters at the beginning of trial?
It turns out Ray is smart and calls at_training_end() no matter what, so I just made a callback and it works now!