# Apple — Theoretical Limits of Language Model Alignment

- Company: Apple (apple.com)
- Announced: 2026-11-12
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/theoretical-limits-alignment
- Record: https://forck.live/items/35426-theoretical-limits-of-language-model-alignment
- Subject: Machine Learning Research

Apple researchers have published a paper characterizing the information-theoretic limits of KL-regularized language model alignment, deriving the maximum expected reward gain possible under a fixed KL-divergence budget. The authors say their first result gives a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the square root of KL used in prior analyses. They reformulate that expression as a covariance under the base model, yielding an estimator that predicts achievable alignment gains from base model samples alone. The paper extends the analysis to proxy rewards, showing the gap between ideal and proxy alignment, known as reward hacking, grows with the magnitude of reward error and when the KL penalty factor decreases. The authors prove that reward ensembling mitigates reward hacking, which they describe as a theoretical justification for a technique used in practice. Empirically, they compute the KL-reward Pareto frontier for two tasks, safety and summarization, and report that best-of-N closely approaches the theoretical limit while PPO and GRPO remain substantially suboptimal.

## Evidence

Verbatim from https://machinelearning.apple.com/research/theoretical-limits-alignment:

> Authors Lucas Monteiro Paes*, Natalie Mackraz*, Barry-John Theobald, Federico Danieli
> 
> Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of- N N alignment, which selects the highest-reward output among N N independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the √ K L √KL used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone.

---

Record: https://forck.live/items/35426-theoretical-limits-of-language-model-alignment
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
