The Asymptotics of Language Model Alignment with Memory

Haricharan Balasundaram; V. Arvind Rameshwar

cs.CLOct 1, 2026

The Asymptotics of Language Model Alignment with Memory

Authors: Haricharan Balasundaram, V. Arvind Rameshwar

Organizations: School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA · IIT Madras · Department of Electrical Engineering, IIT Madras, Chennai, India

Abstract

Language model (LM) alignment broadly aims to perturb a given LM $Q$ into an aligned LM $q$ such that i) the outputs produced by $q$ and $Q$ are 'close' in probability, ii) $q$ has a higher expected reward than $Q$ . Two common techniques for LM alignment are: KL-constrained RL, which requires knowledge of the LM distribution and is computationally expensive, and the best-of- $n$ algorithm, which requires only sampling from the LM. The work of Yang et al. established asymptotic closeness between the distributions produced by the two alignment methods for an $m$ --length i.i.d. token sequence output by the LM, in the limit as $m$ increases to infinity. However, the i.i.d. assumption is not representative of practical LMs, whose output sequences often have memory. In this paper, we extend the asymptotic closeness result to the case when the $m$ --length token sequence outputted by the LM is Markovian. Further, for finite-length output sequences -- particularly, when $m=1$ -- we provide a complete characterization of LM distributions and reward functions for which the KL-divergence between the distributions produced by the two alignment methods is zero -- a question first posed in Yang et al.

Figures & tables

Fig. 1 :

d_{KL}(q_{\beta},q^{\text{best}}_{n,1})

and the theoretical asymptotic rate

\frac{1}{8d^{2}}

d

for

n=10

from Theorem II.7 .

Explore similar work

Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Large Language Model Alignment Fundamental Limits
Apple
Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
Jan 13, 2026Yang Cai, Weiqiang ZhengValue Alignment Test-Time Scaling
Yale University
Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
Sep 28, 2026Florian Eichin, Philipp Mondorf, Andrei Mircea +3Memorization Recurrent Model
MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany

May 8, 2026, cs.LGJ/K move · Enter open · S save

Theoretical Limits of Language Model Alignment

Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald, Federico Danieli

Apple

Large Language Model Alignment Fundamental Limits Divergence Pareto Frontier

Open paper AskPDF arXiv

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-

N

alignment, which selects the highest-reward output among

N

independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the

\sqrt{\texttt{KL}}

used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (reward hacking) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL-reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of-

N

closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.

Abstract

Explore similar work

Theoretical Limits of Language Model Alignment

Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling

Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models