Oh, I just found keep_per_episode_custom_metrics=True, so this solves my problem.
keep_per_episode_custom_metrics=True