Hi everyone, can someone point me to an example/point me in the right direction of how to implement a custom eval function for PBT aka custom metric? Currently using “episode_reward_mean” and want to experiment with different ways to evaluate the performance of the agents in my population. I’m looking for guidance on how to best accomplish this probably from ray tune team.
Hey!
As with other Schedulers, you can define these metrics within your training function.
As a very high level example, if your training function has the following:
loss = #calculate loss
acc = #calculate accuracy
tune.report(loss=loss, acc=acc)
You would be able to pass in loss or acc as your metric in the PBT Scheduler.
“All Trial Schedulers take in a metric , which is a value returned in the result dict of your Trainable and is maximized or minimized according to mode .” Gotcha so i just have to make sure my trainable returns ‘custom_metric’ in the results dict and i should be good. Thanks Matthew!