It seems likely that RLlib does not natively support dynamic, runtime adjustment of the entropy coefficient via a public API, but your approach of patching a setter into the Learner and updating the scheduler’s internal value is a practical workaround. The entropy coefficient in PPO is managed by a Scheduler object per module, and the value is typically set at initialization or via a static schedule, not dynamically at runtime based on arbitrary metrics. Your method of directly modifying the scheduler’s internal state is not officially documented or supported, but it aligns with the internal structure of RLlib’s PPO implementation, where entropy coefficients are stored in entropy_coeff_schedulers_per_module and updated after each training step using the scheduler’s update() method, not via an external setter (see here).
There is no mention in the official sources of a built-in, general-purpose API for dynamically adjusting arbitrary hyperparameters (like entropy coefficient) at runtime based on custom metrics, nor is there a public method for setting the entropy coefficient during training. The only officially supported way to schedule such parameters is via static schedules defined before training starts (see here). Your approach is a reasonable hack, but it is not guaranteed to be stable across RLlib versions. If you want this feature to be officially supported, it may be worth filing a feature request with the Ray team (see related discussion).
Would you like a step-by-step breakdown of how your workaround interacts with RLlib internals and possible alternatives?
Sources:
Hint: Mention @RunLLM in the post for followups.