Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

Kenton Tang; Yuzhu Chen; Fengxiang He

cs.LGFeb 25, 2026

Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

Authors: Kenton Tang, Yuzhu Chen, Fengxiang He

Organizations: University of Edinburgh · University of Science and Technology of China

Abstract

Alignment and adaptation in large language models heavily rely on reinforcement learning from human feedback (RLHF); yet, theoretical understanding of its generalisability remains premature, especially when the learned reward could shift, and the KL control is estimated and clipped. To address this issue, we develop generalisation theory for RLHF that explicitly accounts for (1) reward shift: reward models are trained on preference data from earlier or mixed behaviour policies while RLHF optimises the current policy on its own rollouts; and (2) clipped KL regularisation: the KL regulariser is estimated from sampled log-probability ratios and then clipped for stabilisation, resulting in an error to RLHF. We present generalisation bounds for RLHF, suggesting that the generalisation error stems from a sampling error from prompts and rollouts, a reward shift error, and a KL clipping error. We also discuss special cases of (1) initialising RLHF parameters with a uniform prior over a finite space, and (2) training RLHF by stochastic gradient descent, as an Ornstein-Uhlenbeck process. The theory yields practical implications in (1) optimal KL clipping threshold, and (2) budget allocation in prompts, rollouts, and preference data.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{X},\mathcal{Y}$	Prompt space and response space.
$(x,y)$	A prompt-response pair.
$\rho$	Prompt distribution used for post-training / evaluation.
$\rho_{\mathrm{label}}$	Prompt distribution used for collecting preference data (reward modelling).
$\pi(\cdot\mid x)$	A policy: conditional distribution over responses given prompt $x$ .
$\pi_{\theta}$	Post-trained policy, parameterised by $\theta$ .

Appendix

Table 1: Notation

Symbol	Meaning
$\widehat{J}^{r,\tau}_{n,K}(\theta)$	Empirical objective using $n$ prompts and $K$ rollouts per prompt (reward $r$ , clipping $\tau$ ).
$\widehat{J}^{r,\tau}_{n,\infty}(\theta)$	Conditional (infinite-rollout) analogue: expectation over rollouts given the $n$ sampled prompts.
$n$	Number of sampled prompts.
$K$	Number of rollouts per prompt.
$P$	Prior distribution over $\Theta$ (PAC-Bayes).
$Q$	Posterior distribution over $\Theta$ (PAC-Bayes).

Appendix

Table 2

RL from Human Feedback

Reinforcement learning from human feedback (RLHF) trains AI systems to better match human preferences using comparisons, corrections, and other feedback.

See all 128 papers

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
Sep 11, 2026Kazusato Oko, Annie Ulichney, Nika Haghtalab +1LLM Alignment Preference Alignment
University of California, Berkeley · Center for Advanced Intelligence Project, RIKEN · The Institute of Statistical Mathematics +1
Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
Jul 28, 2026Yunpeng ChuLLM Alignment Reward Shaping
Stony Brook University
A Unifying Lens on Reward Uncertainty in RLHF
Jun 8, 2026Ely Hahami, Yoel Zimmermann, Ray Zhou +1Reward Modeling KL-Regularized RL
Harvard College, Harvard University, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA

Sep 11, 2026J/K move · Enter open · S save

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao

University of California, Berkeley · Center for Advanced Intelligence Project, RIKEN · The Institute of Statistical Mathematics+1

LLM Alignment Preference Alignment RL from Human Feedback

Open paper AskPDF arXiv

While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the distortion -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter

β

when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data (

μ

) and the KL reference policy (

π_{\mathrm{ref}}

). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is

\tildeΘ(βB + β)

, where

B

is an upper bound on the log density ratio between

μ

and

π_{\mathrm{ref}}

. In particular, when there is no distribution mismatch (i.e.,

μ= π_{\mathrm{ref}}

), RLHF achieves the optimal distortion of

O(β)

up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to

μ

Abstract

RL from Human Feedback

Explore similar work

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

A Unifying Lens on Reward Uncertainty in RLHF