1. Severity of the issue: (select one)
Medium: Significantly affects my productivity but can find a workaround.
2. Environment:
Ray version: 2.56.0
Python version: 3.14.6
OS: Linux
Cloud/Infrastructure: SLURM Cluster
Other libs/tools (if relevant): PettingZoo
3. What happened vs. what you expected:
Expected:
Actual:
Hi! I am using Deep RL as a sample-based approximator of what economists might call structural models or agent-based models. The real-life environment I am trying to model has some heterogeneity in terms of how motivated agents are. Some prefer to be absolutely dreadful in some periods in order to get welfare and have the prospect of winning a lot later. Others will never allow themselves to lose a lot.
I think the natural way of encoding this dynamic is to vary discount rates. Ideally, I would be able to train a single policy under various values of a discount rate (which it can observe) drawn from some beta distribution. Different agents would have different discount rates in each episode, and the rates would be re-drawn after an episode is finished.
However, I don’t know whether this would even be possible in Ray RLlib. Any ideas?
It seems likely that you can encode agent heterogeneity in discount rates by making the discount factor (gamma) an observable part of each agent’s observation space, and sampling it per agent per episode. RLlib does not natively support per-agent or per-episode discount rates out of the box, but you can include the discount rate as part of the observation and let the policy learn to condition on it. The actual discounting in RLlib is global (set in the config), so the environment and reward structure must be designed to reflect the agent’s discount preference, or you must implement custom reward processing or value targets if you want true per-agent discounting during training. See discussion in this thread and the multi-agent documentation.
Would you like more detail on how to implement this, or on possible workarounds for per-agent discounting?
I’m not entirely certain of what you’re after, but I did some similar things with RLlib a little while ago.
As far as I can tell, you’re integrating discount rate as a variable within the environment, and intend for the critic’s training to make use of each episode’s specific gamma when calculating value targets. I’ve spent some time hacking at that part of the codebase, and I can’t promise I have the best way, but what I would do is:
Put gamma into your environment’s observations as a do-nothing variable that’s surfaced but has no impact on its behavior.
Create a custom implementation of compute_value_targets that takes a list of gammas rather than a single value.
Override GeneralAdvantageEstimation with a custom version that extracts gammas from the episode’s observations and feeds them into the above.
Create a stub extension of PPO that replaces the base GAE with your version.
Unless I’ve missed something, that should do the trick.