How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. I often see that the optimal “obj” reached by ray-tune does not match the obj I see when I train it with the “optimal” parameters, even after matching random-seeds. If I understood correctly, …

Hi @pchalasani , can you give us more context about the function you’re training and the objective you want to reach? It’s hard to reason about this in an abstract manner. It would also be helpful if you could provide your code with which you don’t currently achieve the expected results, the results …

Thanks @kai for taking a look at this. Some context: I am using Pytorch Lightning, and closely followed the ray/tune docs on how to use it with PTL, with ASHAScheduler and TuneReportCallback, with these params. tune_callback = TuneReportCallback( { "obj": "val_auc", }, …

@kai I just posted another question, which may have some bearing on this.

Thanks for the follow-up question ! Just for others who run into this question, the main reason I suspect after reading your follow-up is that Ray Tune’s final result usually only considers the last reported result, and not the best seen in the run. This can be specified in calls to experiment analys…

@kai I just posted another question relevant to this, with a self-contained minimal example [image] Model initialization is different inside vs outside Ray-Tune Ray Tune How severe does this issue affect your experience of using Ray? High: It blocks me to complete my tas…

Matching ray-tune validation with training validation

Ray Tune

pchalasani March 28, 2022, 1:33am 6

@kai I just posted another question relevant to this, with a self-contained minimal example

Topic		Replies	Views
How does early termination and trial quality evaluation work? Ray Tune	8	1710	May 16, 2024
My training is endless wth tune.run() Ray Tune	8	579	May 3, 2022
Could not find best trial Ray Tune	8	2968	December 21, 2020
Tune.run not executing actual trials Ray Tune	2	485	January 3, 2022
Trial returned a result which did not include the specified metric(s) `val_loss` that `SearchGenerator` expects	0	340	May 24, 2023

Matching ray-tune validation with training validation

Related topics