Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

Siliang Zeng; Luca Viano; Chenliang Li; Jiaxiang Li; Volkan Cevher; Markus Wulfmeier; Stefano Ermon; Alfredo Garcia

cs.LGOct 22, 2026

Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

Authors: Siliang Zeng, Luca Viano, Chenliang Li, Jiaxiang Li, Volkan Cevher, Markus Wulfmeier, Stefano Ermon, Alfredo Garcia,

Organizations: University of Minnesota · EPFL · Texas A&M University · Google DeepMind ·

Abstract

This article provides an introduction to the mathematical foundations and algorithmic frameworks used to align Large Language Models (LLMs) with human intentions, preferences, and values. We discuss standard alignment techniques, such as fine-tuning (SFT), reinforcement learning with human feedback (RLHF), and direct preference optimization (DPO). We also explore the theoretical underpinnings of learning from human preferences, drawing connections to inverse reinforcement learning (IRL) and discrete choice models. We present state-of-the-art algorithms in a tutorial style, discuss their advantages and limitations, and offer insights into practical implementation. Our exposition is intended to serve as a comprehensive resource for researchers and practitioners, providing both a foundational understanding of alignment methodologies and a framework for developing more robust and scalable alignment techniques.

Figures & tables

Figure 1: An illustration of the alignment process.

Figure 2: An illustration of Auto-regressive LLMs. Here <BOS> and <EOS> are two special tokens representing the start and the end of the sentence, respectively. The tokens at the bottom are the input token sequence, and those at the top are the output tokens. In this example, the token sequence “<BOS> All transformer models ” is given to the LLM and “are generative models <EOS>” is the prediction. Each of the previous tokens will be used to predict the next token, corresponding to ( 1 ).

Figure 3: An illustration of the standard LLM Alignment process. The standard RLHF pipeline first trains a supervised model on demonstrations, then builds a reward model from preference data, and finally optimizes the policy with RL guided by the reward model.

Figure 4: True distribution in blue and estimated distribution in green. When the estimated distribution has a smaller support than the true one, as in the left plot, we say that it has a mode seeking. Vice versa, the estimated distribution has a mode covering behavior if its support is larger, as in the right plot. Image from https://sander.ai/2020/03/24/audio-generation.html .

\widehat{\ell}_{\mathrm{SPIN}}(\theta):=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{demo}},\;y^{\prime}\sim\pi(\cdot\mid x;\theta_{t})}\left[\ell\left(\beta\log\left(\frac{\pi(y\mid x;\theta)}{\pi(y\mid x;\theta_{t})}\right)-\beta\log\left(\frac{\pi(y^{\prime}\mid x;\theta)}{\pi(y^{\prime}\mid x;\theta_{t})}\right)\right)\right].

Protocol 2 SFT (Supervised Fine Tuning) via SPIN (Self-Play fIne tuNing)

\min_{\theta}\max_{\phi}\widehat{\ell}_{\mathrm{GAIL}}(\theta,\phi):=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{demo}},\;y^{\prime}\sim\pi(\cdot\mid x;\theta)}\Big[\log(1-D_{\phi}(x,y))+\log(D(x,y^{\prime}))+\lambda\log\pi_{\theta}(y^{\prime}|x)\Big].

Protocol 3 GAIL (Generative Adversarial Imitation Learning) [ 59 ]

\phi^{\star}=\argmin_{\phi\in\Phi}\ \widehat{\ell}_{\rm RM}(\phi):=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\rm pref}}\Big[\log\Big(\sigma\big(r(x^{i},y^{i}_{w};\phi)-r(x^{i},y^{i}_{l};\phi)\big)\Big)\Big].

Protocol 4 RLHF (Reinforcement Learning from Human Feedback)

	$\displaystyle\widehat{\ell}_{\rm PPO}(\theta)$	$\displaystyle=-\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{prompts}},y\sim\pi(\cdot\|x;\theta_{t})}\Bigg[\frac{1}{\|y\|}\sum_{h=1}^{\|y\|}\Bigg\{\min\Bigg[\frac{\pi(y_{h}\|x,y_{<h};\theta)}{\pi(y_{h}\|x,y_{<h};\theta_{t})}\hat{A}_{h},$
		$\displaystyle\quad\quad\quad\quad\text{clip}\Bigg(\frac{\pi(y_{h}\|x,y_{<h};\theta)}{\pi(y_{h}\|x,y_{<h};\theta_{t})},1-\epsilon,1+\epsilon\Bigg)\hat{A}_{h}\Bigg]\Bigg\}\Bigg]$

Protocol 5 PPO (Proximal Policy Optimization) [ 47 ]

\displaystyle\nabla\widehat{\ell}_{\rm REINFORCE}(\theta_{t})

\displaystyle=-\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{prompts}},y\sim\pi(\cdot|x;\theta_{t})}\big[r(x,y)\nabla_{\theta}\log\pi(y|x;\theta_{t})\big]

Protocol 6 REINFORCE [ 58 ]

\hat{A}_{j}=r^{j}-\frac{1}{G-1}\sum_{i\neq j}r^{i}.

Protocol 7 RLOO (REINFORCE Leave-One-Out) [ 2 ]

\hat{A}^{j}=\frac{r^{j}-\operatorname*{mean}(\{r^{j}\}_{j=1}^{G})}{\operatorname*{std}(\{r^{j}\}_{j=1}^{G})},\quad j=1,\cdots,G

Protocol 8 GRPO (Group Relative Policy Optimization) [ 48 ]

Figure 5: Estimated memory consumption of running the Pythia-1B model on TL;DR dataset with batch-size 2 on a single device

Action	$y_{1}$	$y_{2}$	$y_{3}$
$\pi_{\mathrm{ref}}$	$0.5$	$0.5$	$0$
$D_{\mathrm{pref}}$	$\{(y_{w}=y_{1},y_{l}=y_{2})\}$
$\pi_{\mathrm{DPO}}$	$\alpha$	$0.0$	$1-\alpha$
$\pi_{\mathrm{RLHF}}$	1	0	0

Table 1: The policy

\pi_{\mathrm{DPO}}

minimizes

\ell_{\mathrm{DPO}}

for any

\alpha\in[0,1)

. Hence, for

\alpha>0

, it produces a policy with support which is not a subset of

\pi_{\mathrm{ref}}

. The same can not happen for

\pi_{\mathrm{RLHF}}

that minimizes

\ell_{\mathrm{RLHF}}

\widehat{\ell}_{\mathrm{DPO}}(\theta):=-\mathbb{E}_{(x,y_{w},y_{l})\in\mathcal{D}_{\rm pref}}\left[{\log\left({\sigma\left({\beta\log\left({\frac{\pi(y_{w}|x;\theta)}{\pi_{\mathrm{ref}}(y_{w}|x)}}\right)-\beta\log\left({\frac{\pi^{\star}(y_{l}|x;\theta)}{\pi_{\mathrm{ref}}(y_{l}|x)}}\right)}\right)}\right)}\right]

Protocol 9 DPO (Direct Preference Optimization)

	PPO Win	Tie	DPO Win
PPO V.S. DPO	42	28	30

Table 2: On HH-RLHF, GPT-4 is used to decide the winner between PPO and DPO models’ output. Taken from [ 67 ] .

\phi^{\star}=\argmin_{\phi\in\Phi}\ \widehat{\ell}_{\rm PM}(\phi):=-\mathbb{E}_{(x,y_{w},y_{l})\in\mathcal{D}_{\rm pref}}\log\Big(\mathcal{P}(y_{w}\succ y_{l}|x;\phi)\Big).

Protocol 10 NLHF (Nash Learning from Human Feedback)

Algorithm	Needs Reward?	Dataset Type	Needs Value Fn.?
SFT	✗	Demonstration	✗
SPIN	✗	Demonstration + self-generated	✗
IRL	✓	Demonstration	✗
REINFORCE	✓	Demonstration + Preference	✗
RLOO	✓	Demonstration + Preference	✗
PPO	✓	Demonstration + Preference	✓

Table 3: Comparison of alignment algorithms by reward requirement, dataset type, and value-function (critic) dependence. Here since AIHF can use different kinds of RL algorithms such as PPO and DPO, we denote it as ‘flexible’ in the last column.

Panel of Figure 6 — Figure 6: Left: Evaluation of policy models on the Open LLM Leaderboard. Right: Evaluation of reward models using RewardBench, where reward accuracy measures the proportion of preference pairs in which the reward model assigns a higher score to the preferred response. Results show that high-quality demonstration datasets can effectively enhance both reward and policy models via the IRL method.

Tasks	First turn	Second turn	Average
mistral-7b-sft-beta	5.66	5.09	5.37
SPIN-Iter1	6.75	5.56	6.16
SPIN-Iter2	3.18	3.41	3.29
IRL-Iter1-Policy	6.71	5.96	6.33
IRL-Iter2-Policy	7.01	6.19	6.60

Table 4: Evaluation of Policy Models in MT-Bench. The results show that the IRL method can effectively improve the quality of policy models on multi-turn questions.

Figure 7: Performance comparison of alignment methods. The AIHF-trained model demonstrates superior alignment across various downstream tasks.

Reward Model	Chat	Chat Hard	Safety	Reasoning	Average
DPO Reward Model	37.43%	55.92%	64.14%	47.33%	51.21%
BTL Reward Model	95.11%	56.58%	63.69%	69.22%	71.15%
AIHF Reward Model	94.41%	55.37%	63.98%	76.75%	72.63%

Table 5: Evaluation of reward models in Reward-Bench. It can be seen that the reward model estimated by AIHF which incorporates both demonstrations and preference achieves better performance.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

Apr 2, 2026

Reinforcement Learning from Human Feedback: A Statistical Perspective

Reinforcement learning from human feedback (RLHF) has emerged as a central framework for aligning large language models (LLMs) with human preferences. Despite its practical success, RLHF raises fundamental statistical questions because it relies on noisy, subjective, and often heterogeneous feedback to learn reward models and optimize policies. This survey provides a statistical perspective on RLHF, focusing primarily on the LLM alignment setting. We introduce the main components of RLHF, including supervised fine-tuning, reward modeling, and policy optimization, and relate them to familiar statistical ideas such as Bradley-Terry-Luce (BTL) model, latent utility estimation, active learning, experimental design, and uncertainty quantification. We review methods for learning reward functions from pairwise preference data and for optimizing policies through both two-stage RLHF pipelines and emerging one-stage approaches such as direct preference optimization. We further discuss recent extensions including reinforcement learning from AI feedback, inference-time algorithms, and reinforcement learning from verifiable rewards, as well as benchmark datasets, evaluation protocols, and open-source frameworks that support RLHF research. We conclude by highlighting open challenges in RLHF. An accompanying GitHub demo https://github.com/Pangpang-Liu/RLHF_demo illustrates key components of the RLHF pipeline.

Pangpang Liu, Chengchun Shi, Will Wei Sun

Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.

RL from Human Feedback LLM Alignment Reward Modeling Preference Learning

Ask PDF

Jul 3, 2026

Unbiased Alignment for Large Language Models with Noisy Preferences

The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.

Jialiang Wang, Xianming Liu, Xiong Zhou +2

1Harbin Institute of Technology · 2City University of Hong Kong.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Learning with Noisy Labels Preference Learning

Ask PDF

May 26, 2026

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases

Reinforcement Learning from Human Feedback (RLHF) is the standard method to align Large Language Models (LLMs) with human preferences. In this work, we introduce alignment tampering, a potential vulnerability where the LLM undergoing alignment influences the preference dataset, causing RLHF to amplify undesired behaviors. This arises from core limitations of RLHF: (1) preference datasets are constructed from the LLM's own outputs, allowing it to influence them, and (2) pairwise comparisons only indicate which response is better, not why. These limitations can be exploited to cause alignment tampering. For example, if an LLM generates biased responses with higher quality, annotators will prefer them based on quality. However, preference labels do not distinguish quality from bias, and the reward model inherits this limitation. Optimizing such rewards through reinforcement learning or best-of-N sampling can amplify misaligned biases. Our experiments demonstrate amplification across diverse biases: from keyword bias to propaganda (e.g., sexism), brand promotion, and instrumental goal-seeking. Mitigation remains challenging, as existing techniques for robust RLHF fail to fully resolve alignment tampering without sacrificing response quality. These findings reveal structural vulnerabilities of current RLHF and emphasize the need to prevent this vulnerability. Project page: https://alignment-tampering.github.io/

Dongyoon Hahm, Dylan Hadfield-Menell, Kimin Lee

1KAIST · 2MIT

LLM Alignment Data Poisoning Attacks Value Alignment RL from Human Feedback

Ask PDF

Abstract

LLM Alignment

Explore similar work

Reinforcement Learning from Human Feedback: A Statistical Perspective

Unbiased Alignment for Large Language Models with Noisy Preferences

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases