Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards

Huiming Zhang; Binghan Li; Wan Tian; Qiang Sun

stat.MLApr 12, 2026

Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards

Authors: Huiming Zhang, Binghan Li, Wan Tian, Qiang Sun

Organizations: BUAA and BAIC-FBPC · BUAA · PKU · UofT and MBZUAI

Abstract

Classical information-theoretic learning bounds typically rely on KL mutual information and moment-generating-function (MGF) arguments, which are well matched to bounded or sub-Gaussian losses but can be ineffective when losses or rewards are heavy-tailed. We develop a tail-aware information-theoretic framework for sub-Weibull data, where the tail parameter $θ$ controls the tail heaviness: $θ=2$ corresponds to sub-Gaussian, $θ=1$ to sub-exponential, and $0<θ<1$ to genuinely heavy tails. Our key technical ingredient is a decorrelation lemma that bounds change-of-measure expectations using a shifted-log $f_θ$ -divergence, which admits explicit comparisons to Rényi divergence without MGF arguments. On the empirical-process side, we establish sharp maximal inequalities and a Dudley-type chaining bound for sub-Weibull processes, with logarithmic and entropy terms raised to the power $1/θ$ . These tools yield tail-adaptive selection bounds and a multiscale information-theoretic Dudley inequality based on shifted-log and Rényi mutual information. We apply our theory to large language models (LLMs) in the context of reward hacking within reinforcement learning from human feedback (RLHF). We show that Rényi-regularized alignment provides finite reward guarantees and ensures that best-of-N policies remain well-controlled, thereby mitigating the catastrophic Goodhart effects where standard KL-regularization fails. We illustrate Rényi-regularized RLHF by experiments, including controlled heavy-tailed rewards and token-space reward attacks.

Figures & tables

Object	Classical	This paper
Finite maxima	$\mathbb{E}\max_{i}X_{i}\lesssim\sqrt{\log n}$	$\mathbb{E}\max_{i}X_{i}\lesssim(\log n)^{1/\theta}$ (Lemma 2.1 )
Process suprema	Dudley: $\int\sqrt{\log N_{\varepsilon}}\,d\varepsilon$	Sub-Weibull Dudley: $\int(\log N_{\varepsilon})^{1/\theta}\,d\varepsilon$ (Theorem 2.2 )
Data-dependent selection	KL mutual information $\sqrt{I(W;S)}$	Shifted-log $I_{f_{\theta}}(W;S)$ and Rényi refinements (Theorem 4.2 )
Multiscale selection	Sub-Gaussian information chaining	Sub-Weibull chaining with $I_{f_{\theta}}([W]_{k};S)$ and Rényi refinements (Theorem 4.3 )
RLHF trust regions	KL trust regions can fail under heavy tails	Rényi trust regions control reward inflation (Theorem 5.2 )
Best-of- $n$ alignment	Light-tailed maximal control	Rényi divergence gives $(\log n)^{1/\theta}$ -type reward control (Theorem 5.3 )

Table 1: Roadmap of results.

\theta

is the sub-Weibull tail parameter and

N_{\varepsilon}=N(T,d,\varepsilon)

$\varepsilon$	$1/20$	$1/30$	$1/40$	$1/50$	$1/100$	$1/200$	$1/400$
MI	$\infty$	$\infty$	$\infty$	$\infty$	$\infty$	$\infty$	$\infty$
CM	832.01	832.01	832.01	832.01	832.01	832.01	832.01
Rényi CMI	71.93	71.46	71.30	71.22	71.12	71.10	71.09
$-\mathbb{E}[X_{W}]$	0.180	0.120	0.090	0.072	0.036	0.018	0.009

Table 2:

-\mathbb{E}[X_{W}]

and upper bounds for

\theta=0.5

. MI: Theorem 4.2 ; CM: Theorem 2.2 ; Rényi CMI: Theorem 4.3 .

Figure 1: Empirical distributions and QQ plots of proxy reward model scores before and after the power transformation for

\kappa=8

. The upper panels show the empirical distributions of the original reward

r(x,y)

and the transformed reward

\widetilde{r}(x,y)=\operatorname{sign}(r(x,y))|r(x,y)|^{8}

under

\beta=1

with

\alpha=1

(KL) and

\alpha=2

(Rényi). The original rewards are overlaid with fitted normal density curves, whereas the transformed rewards are overlaid with kernel density estimates. The lower panels report the corresponding QQ plots against the normal reference distribution. The transformation preserves reward ordering while substantially amplifying the heavy-tailedness of the training signal.

Figure 2: Reward–divergence relationship for

\alpha=1

(KL) with

\beta=1

under different power exponents

\kappa

. The four subplots correspond to

\kappa=1,6,8,10

, ordered from left to right and top to bottom. As

\kappa

increases, a clearer separation emerges between proxy and gold rewards: the proxy reward tends to keep increasing with divergence, whereas the gold reward exhibits a rise-then-collapse pattern, indicating over-optimization under heavy-tailed proxy rewards.

Figure 3: Reward–divergence relationship under Rényi regularization with

\beta=1

, for orders

\alpha\in\{2,\ldots,10\}

and different power exponents

\kappa

. The four subplots correspond to

\kappa=1,6,8,10

, ordered from left to right and top to bottom. Across

\alpha\in\{2,\ldots,10\}

, the gold reward is substantially more stable and typically increases before leveling off, rather than exhibiting the sharp rise-then-collapse pattern observed under KL, suggesting that Rényi regularization provides a more reliable trust-region signal in the heavy-tailed regime.

Figure 4: Proxy–proxy-gold reward relationship under

\beta=1

, for Rényi orders

\alpha\in\{1,\ldots,10\}

and power exponents

\kappa\in\{1,6,8,10\}

. The four subplots correspond to

\kappa=1,6,8,10

, ordered from left to right and top to bottom. As

\kappa

increases, the KL case (

\alpha=1

) becomes clearly non-monotone, with the gold reward eventually declining as the proxy reward continues to improve, whereas the curves for

\alpha\geq 2

remain smoother and largely monotone increasing, indicating a more reliable coupling between proxy and gold objectives.

Figure 5: Tail behavior of reward distributions under TOMPA token-space attacks with KL and Rényi regularization. The top row shows KL regularization, while the middle and bottom rows correspond to Rényi regularization with

\alpha=2

and

\alpha=4

, respectively. Left panels plot the empirical MGF

M(t)

against the standard normal benchmark. KL regularization exhibits stronger right-tail inflation and upper-tail deviations, while Rényi regularization attenuates these tail effects, especially when

\alpha=4

Figure 6: Reward–divergence diagnostics under TOMPA token-space attacks with KL and Rényi regularization. The upper panel reports the relationship between reward and divergence for

\alpha=1

(KL),

\alpha=2,4

(Rényi), where points denote observed rewards and curves denote smoothed trends. The lower panel compares the directly optimized target reward with the auxiliary gold reward, obtained by decoding the mapped response IDs and scoring the resulting answer-only text with an external frozen reward model. Under KL regularization, the target reward increases as divergence grows, whereas the gold reward first increases and then decreases. Under Rényi regularization, both rewards tend to increase together and gradually stabilize.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Partition-cell chaining argument.

Figure 8: Empirical MGFs comparison for the original and transformed reward. Left:

\beta=1

with

\alpha=1

(KL). Right:

\beta=1

with

\alpha=2

(Rényi). Blue curves correspond to the original reward

r(x,y)

, and red curves correspond to the transformed reward

\widetilde{r}(x,y)=\operatorname{sign}(r(x,y))\,|r(x,y)|^{8}

. The much faster growth of the empirical MGF for the transformed rewards is consistent with stronger tail amplification.

Figure 9: Reward–divergence relationship under the centered Weibull-noise reward with shape parameters 0.2, 0.3, 0.4, and 0.5, ordered from left to right and top to bottom. Across all Weibull shape parameters, the reward–divergence curves remain stable: the gold reward typically increases and then levels off, rather than exhibiting a sharp collapse, indicating that higher-order Rényi regularization provides a reliable trust-region with

\beta=1

Figure 10: Proxy–proxy-gold reward relationship under the Weibull-noise reward construction with

\beta=1

and shape parameters 0.2 to 0.5, ordered from left to right and top to bottom. Across all noise levels, the proxy–gold curves remain monotone increasing over the observed range, showing that improvements in the proxy reward continue to translate consistently into improvements in the gold reward even under heavier-tailed Weibull.

Information-Theoretic Generalization Bounds

Information-theoretic generalization bounds use information measures to relate training-data dependence to the gap between training and test performance.

See all 22 papers

Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Reward Hacking LLM Alignment
Apple
Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
Feb 25, 2026Kenton Tang, Yuzhu Chen, Fengxiang HeRL from Human Feedback Generalization in RL
University of Edinburgh · University of Science and Technology of China
Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
Feb 20, 2026Johannes Ackermann, Michael Noukhovitch, Takashi Ishida +1Reward Hacking Reinforcement Learning with Verifiable Rewards
The University of Tokyo · Mila · RIKEN AIP

May 8, 2026J/K move · Enter open · S save

Theoretical Limits of Language Model Alignment

Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald, Federico Danieli

Apple

Reward Hacking LLM Alignment KL-Regularized RL Preference Alignment

Open paper AskPDF arXiv

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-

N

alignment, which selects the highest-reward output among

N

independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the

\sqrt{\texttt{KL}}

used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (reward hacking) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL-reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of-

N

closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.

Abstract

Information-Theoretic Generalization Bounds

Explore similar work

Theoretical Limits of Language Model Alignment

Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards