Confused about "adjust_nstep"

fen13718010069 · August 12, 2023, 7:38am

Hi, I’m confused about the implementation in multi step reward(adjust_nstep):

ray-project/ray/blob/f82880eda18dced7ea396d0dcced9246909de79f/rllib/evaluation/postprocessing.py#L19


      
          from ray.rllib.utils.typing import AgentID
          
          
          class Postprocessing:
              """Constant definitions for postprocessing."""
          
              ADVANTAGES = "advantages"
              VALUE_TARGETS = "value_targets"
          
          
          def adjust_nstep(n_step: int, gamma: float, batch: SampleBatch) -> None:
              """Rewrites `batch` to encode n-step rewards, dones, and next-obs.
          
              Observations and actions remain unaffected. At the end of the trajectory,
              n is truncated to fit in the traj length.
          
              Args:
                  n_step: The number of steps to look ahead and adjust.
                  gamma: The discount factor.
                  batch: The SampleBatch to adjust (in place).

Here, len_ and n_step is euqal. so except for the first trajectory in the batch, the multi-step returns for the remaining trajectories would be smaller than nstep. Is my understanding correct ? Is this implementaion reasonable?

Topic		Replies	Views
How is the n-step DQN implemented in rllib? RLlib	2	481	January 18, 2022
Semi-MDPs and RLlib: Problems where times to execute an action strongly vary RLlib	2	186	August 4, 2021
Post process trajectory with full episode RLlib	1	339	October 17, 2023
'timesteps_per_iteration' parameter RLlib	1	666	July 21, 2021
Last step not counted in off policy estimation RLlib	3	337	December 8, 2020

Confused about "adjust_nstep"

Related Topics