Understanding the basics of PPOTrainer

RLlib
1

If the below code is
agent = PPOTrainer(config, env=“CartPole-v1”) -----------------------------------(1)

for _ in range(1):
result = agent.train() -------------------------------------------------------------(2)

Does (1) mean, that it has collected the training data of default 4000 batch size ( called the reset, step functions in the environment )

Does (2) mean, that with the data already collected from (1) , the policy is getting changed or trained based on loss calculated ?

I see so many versions of implementation of the above and getting confused.
example:

algo = (
PPOConfig()
.rollouts(num_rollout_workers=1)
.resources(num_gpus=0)
.environment(env=“CartPole-v1”)
.build()
)

for i in range(10):
result = algo.train()

isnt this the same ?

2

Hi @Archana_R

(1) builds the training object in this case the PPO RL algorithm. No interaction with the environment occurs here other than to build them (and perhaps call reset, I am not sure if this happens here or in 2).

(2) for each call to train it will:
(a) sample 4000 new steps from the environment using the current version of the policy
(b) train on-policy with the trajectories that were sampled in (a)

3

Thanks this helps . I have 1 more basic question.
1 step implies → 1 iteration in visiting step function with a chosen action ?

Also, what do you think about the last statement on the different versions of implementation. ?