From the source
Apple researchers have published a paper characterizing the information-theoretic limits of KL-regularized language model alignment, deriving the maximum expected reward gain possible under a fixed KL-divergence budget.
- The authors say their first result gives a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the square root of KL used in prior analyses.
- They reformulate that expression as a covariance under the base model, yielding an estimator that predicts achievable alignment gains from base model samples alone.
- The paper extends the analysis to proxy rewards, showing the gap between ideal and proxy alignment, known as reward hacking, grows with the magnitude of reward error and when the KL penalty factor decreases.
- The authors prove that reward ensembling mitigates reward hacking, which they describe as a theoretical justification for a technique used in practice.
- Empirically, they compute the KL-reward Pareto frontier for two tasks, safety and summarization, and report that best-of-N closely approaches the theoretical limit while PPO and GRPO remain substantially suboptimal.
