How severe does this issue affect your experience of using Ray?
Low: It annoys or frustrates me for a moment.
I use a lot of custom metrics to monitor the learning performance. TensorBoard takes ages to load all the information from the server my code is working on and to draw the plots. I want to write some simple functions to save relevant plots as pictures on the server and then download them. The problem is I cannot find where the custom metrics are stored. The progress.csv file contains the default metrics only.
Recently, I stumbled across the topic of custom metrics. Apart from your concern of “Where are they stored”, I also see the challenge of accessing the values per iteration and saving thoses for later reporting. But coming back to your point, you may have a look at the file result.json which appears in the same folder as the progress.csv.
Thank you for the reply. Indeed, the result.json file contains the “custom_metrics” field. However, this field became empty after I upgraded from Ray 2.4.0 to Ray 2.37.0. In the latter version, the progress.csv file contains custom metrics if I run the code on my MacBook and does not have any custom metrics if I run the same code on a cluster. But this is a different problem.