You do not need to manually accumulate Worker rewards; RLlib automatically accumulates rewards for each agent between their actions, including in hierarchical RL. Any rewards given to an agent when it is not acting are stored and summed until the agent acts again, at which point the total is attributed to the last action. See multi-agent reward handling discussion and multi-agent episode code.
Would you like more detail on how this works internally?
Sources:
Hint: Mention @Herald in the post for followups.