Matching ray-tune validation with training validation

[pchalasani]

@kai I just posted another question relevant to this, with a self-contained minimal example

Model initialization is different inside vs outside Ray-Tune

How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. One of the big hopes with Ray-tune is that once it finds a good hparam config C with metric result X, one can use the config C to do normal training outside of ray-tune and (after ensuring identical seeds and data inputs) get an identical training run to what occurred inside the ray-tune trial. The actual final metric may differ somewhat from X because of differences in early termination a…