Metrics Reporting Frequency

RLlib
1

Hi there,

I want to do the following:

  • Collect multiple episodes ("batch_mode": "complete_episodes") by workers and then
  • show metrics in Tensorboard in Episodes instead of rollout_fragment_lengths. FYI: I have Episodes of fixed length of 1,000ts.

I have researched a little bit in the TBXLoggerCallback and the CollectMetrics classes, but have found so far no solution. Has anyone a hint for me?

@mannyv, @sven1977 maybe?

To show you what I mean:
image

I ran 2 episodes with rollout_fragment_length=1000 (horizon=1000) and with 2 workers. As a result I get a single point in Tensorboard for the single train iteration that used 2000 timesteps.

What I want to have is this:
image

no matter if I train with a single worker or 2 workers. This would be possible, if the frequency in Tensorboard would be based on episodes instead of training iterations.

2

I think right now we only support logging on the axis of number of sampling timesteps instead of training/sampling episodes since it is a more global/less confusing measurement of agents for users when they’re doing comparisons across different trained agents.

3

Hi @avnishn,

I agree on the more global measurement of training process via timesteps instead of episodes.

In my case I would still like to have timesteps, though, however, I would like to have the measurement after the amount of timesteps that have been taking in each episode.

The reason for this is that the episode measurements are quite important and as soon as I start to collect more than 1 episode by a worker or use multiple workers, I loose information about the episode results (results after 1000 timesteps).