Get the number of training steps when loading a trained agent

RLlib
1

I train and save my RL agents (using PPO) and later load them again for inference using agent = restore(path).

Is there any way to get the number of steps the loaded agent was trained? I’d like to print/show them in my application.
I saw there is agent.training_iteration and agent.config, which holds a ton of config settings, but I can’t seem to find the number of training steps.

I also don’t think I can easily derive it from the number of training iterations. In my example, the agent was trained for 90 iterations with a batch size defaulting to 4000. Still, my results show that the agent was trained for 5600 steps per iteration, resulting in 504000 training steps.
I don’t understand where the 5600 comes from and also don’t find it anywhere in the configuration.

2

Hm, it seems like the training steps is really just num_training_iterations * batch_size.
Not sure, why my batch size was 5600 in the one case; in all other cases it was the default of 4000 and the number of train steps added up.

I guess this answers my own question.

3

Hm, it seems to depend on the machine. If I run my RLlib agent on a specific machine, it always defaults to 5600 train steps per iteration rather than the expected 4000. No idea why.

Is it possible that config['num_workers'] also somehow affects the number of train steps per iteration? I checked and batch size, rollout fragment length, sgd size, etc are the same for all of my machines. But the machine with 5600 instead of 4000 train steps per iteration has fewer CPU cores and only uses 14 instead of 20 workers.