Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

Siliang Zeng; Luca Viano; Chenliang Li; Jiaxiang Li; Volkan Cevher; Markus Wulfmeier; Stefano Ermon; Alfredo Garcia

cs.LGOct 22, 2026

Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

Authors: Siliang Zeng, Luca Viano, Chenliang Li, Jiaxiang Li, Volkan Cevher, Markus Wulfmeier, Stefano Ermon, Alfredo Garcia,

Organizations: University of Minnesota · EPFL · Texas A&M University · Google DeepMind ·

Abstract

This article provides an introduction to the mathematical foundations and algorithmic frameworks used to align Large Language Models (LLMs) with human intentions, preferences, and values. We discuss standard alignment techniques, such as fine-tuning (SFT), reinforcement learning with human feedback (RLHF), and direct preference optimization (DPO). We also explore the theoretical underpinnings of learning from human preferences, drawing connections to inverse reinforcement learning (IRL) and discrete choice models. We present state-of-the-art algorithms in a tutorial style, discuss their advantages and limitations, and offer insights into practical implementation. Our exposition is intended to serve as a comprehensive resource for researchers and practitioners, providing both a foundational understanding of alignment methodologies and a framework for developing more robust and scalable alignment techniques.

Figures & tables

Figure 1: An illustration of the alignment process.

Figure 2: An illustration of Auto-regressive LLMs. Here <BOS> and <EOS> are two special tokens representing the start and the end of the sentence, respectively. The tokens at the bottom are the input token sequence, and those at the top are the output tokens. In this example, the token sequence “<BOS> All transformer models ” is given to the LLM and “are generative models <EOS>” is the prediction. Each of the previous tokens will be used to predict the next token, corresponding to ( 1 ).

Figure 3: An illustration of the standard LLM Alignment process. The standard RLHF pipeline first trains a supervised model on demonstrations, then builds a reward model from preference data, and finally optimizes the policy with RL guided by the reward model.

Figure 4: True distribution in blue and estimated distribution in green. When the estimated distribution has a smaller support than the true one, as in the left plot, we say that it has a mode seeking. Vice versa, the estimated distribution has a mode covering behavior if its support is larger, as in the right plot. Image from https://sander.ai/2020/03/24/audio-generation.html .

\widehat{\ell}_{\mathrm{SPIN}}(\theta):=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{demo}},\;y^{\prime}\sim\pi(\cdot\mid x;\theta_{t})}\left[\ell\left(\beta\log\left(\frac{\pi(y\mid x;\theta)}{\pi(y\mid x;\theta_{t})}\right)-\beta\log\left(\frac{\pi(y^{\prime}\mid x;\theta)}{\pi(y^{\prime}\mid x;\theta_{t})}\right)\right)\right].

Protocol 2 SFT (Supervised Fine Tuning) via SPIN (Self-Play fIne tuNing)

\min_{\theta}\max_{\phi}\widehat{\ell}_{\mathrm{GAIL}}(\theta,\phi):=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{demo}},\;y^{\prime}\sim\pi(\cdot\mid x;\theta)}\Big[\log(1-D_{\phi}(x,y))+\log(D(x,y^{\prime}))+\lambda\log\pi_{\theta}(y^{\prime}|x)\Big].

Protocol 3 GAIL (Generative Adversarial Imitation Learning) [ 59 ]

\phi^{\star}=\argmin_{\phi\in\Phi}\ \widehat{\ell}_{\rm RM}(\phi):=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\rm pref}}\Big[\log\Big(\sigma\big(r(x^{i},y^{i}_{w};\phi)-r(x^{i},y^{i}_{l};\phi)\big)\Big)\Big].

Protocol 4 RLHF (Reinforcement Learning from Human Feedback)

	$\displaystyle\widehat{\ell}_{\rm PPO}(\theta)$	$\displaystyle=-\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{prompts}},y\sim\pi(\cdot\|x;\theta_{t})}\Bigg[\frac{1}{\|y\|}\sum_{h=1}^{\|y\|}\Bigg\{\min\Bigg[\frac{\pi(y_{h}\|x,y_{<h};\theta)}{\pi(y_{h}\|x,y_{<h};\theta_{t})}\hat{A}_{h},$
		$\displaystyle\quad\quad\quad\quad\text{clip}\Bigg(\frac{\pi(y_{h}\|x,y_{<h};\theta)}{\pi(y_{h}\|x,y_{<h};\theta_{t})},1-\epsilon,1+\epsilon\Bigg)\hat{A}_{h}\Bigg]\Bigg\}\Bigg]$

Protocol 5 PPO (Proximal Policy Optimization) [ 47 ]

\displaystyle\nabla\widehat{\ell}_{\rm REINFORCE}(\theta_{t})

\displaystyle=-\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{prompts}},y\sim\pi(\cdot|x;\theta_{t})}\big[r(x,y)\nabla_{\theta}\log\pi(y|x;\theta_{t})\big]

Protocol 6 REINFORCE [ 58 ]

\hat{A}_{j}=r^{j}-\frac{1}{G-1}\sum_{i\neq j}r^{i}.

Protocol 7 RLOO (REINFORCE Leave-One-Out) [ 2 ]

\hat{A}^{j}=\frac{r^{j}-\operatorname*{mean}(\{r^{j}\}_{j=1}^{G})}{\operatorname*{std}(\{r^{j}\}_{j=1}^{G})},\quad j=1,\cdots,G

Protocol 8 GRPO (Group Relative Policy Optimization) [ 48 ]

Figure 5: Estimated memory consumption of running the Pythia-1B model on TL;DR dataset with batch-size 2 on a single device

Action	$y_{1}$	$y_{2}$	$y_{3}$
$\pi_{\mathrm{ref}}$	$0.5$	$0.5$	$0$
$D_{\mathrm{pref}}$	$\{(y_{w}=y_{1},y_{l}=y_{2})\}$
$\pi_{\mathrm{DPO}}$	$\alpha$	$0.0$	$1-\alpha$
$\pi_{\mathrm{RLHF}}$	1	0	0

Table 1: The policy

\pi_{\mathrm{DPO}}

minimizes

\ell_{\mathrm{DPO}}

for any

\alpha\in[0,1)

. Hence, for

\alpha>0

, it produces a policy with support which is not a subset of

\pi_{\mathrm{ref}}

. The same can not happen for

\pi_{\mathrm{RLHF}}

that minimizes

\ell_{\mathrm{RLHF}}

\widehat{\ell}_{\mathrm{DPO}}(\theta):=-\mathbb{E}_{(x,y_{w},y_{l})\in\mathcal{D}_{\rm pref}}\left[{\log\left({\sigma\left({\beta\log\left({\frac{\pi(y_{w}|x;\theta)}{\pi_{\mathrm{ref}}(y_{w}|x)}}\right)-\beta\log\left({\frac{\pi^{\star}(y_{l}|x;\theta)}{\pi_{\mathrm{ref}}(y_{l}|x)}}\right)}\right)}\right)}\right]

Protocol 9 DPO (Direct Preference Optimization)

	PPO Win	Tie	DPO Win
PPO V.S. DPO	42	28	30

Table 2: On HH-RLHF, GPT-4 is used to decide the winner between PPO and DPO models’ output. Taken from [ 67 ] .

\phi^{\star}=\argmin_{\phi\in\Phi}\ \widehat{\ell}_{\rm PM}(\phi):=-\mathbb{E}_{(x,y_{w},y_{l})\in\mathcal{D}_{\rm pref}}\log\Big(\mathcal{P}(y_{w}\succ y_{l}|x;\phi)\Big).

Protocol 10 NLHF (Nash Learning from Human Feedback)

Algorithm	Needs Reward?	Dataset Type	Needs Value Fn.?
SFT	✗	Demonstration	✗
SPIN	✗	Demonstration + self-generated	✗
IRL	✓	Demonstration	✗
REINFORCE	✓	Demonstration + Preference	✗
RLOO	✓	Demonstration + Preference	✗
PPO	✓	Demonstration + Preference	✓

Table 3: Comparison of alignment algorithms by reward requirement, dataset type, and value-function (critic) dependence. Here since AIHF can use different kinds of RL algorithms such as PPO and DPO, we denote it as ‘flexible’ in the last column.

Panel of Figure 6 — Figure 6: Left: Evaluation of policy models on the Open LLM Leaderboard. Right: Evaluation of reward models using RewardBench, where reward accuracy measures the proportion of preference pairs in which the reward model assigns a higher score to the preferred response. Results show that high-quality demonstration datasets can effectively enhance both reward and policy models via the IRL method.

Tasks	First turn	Second turn	Average
mistral-7b-sft-beta	5.66	5.09	5.37
SPIN-Iter1	6.75	5.56	6.16
SPIN-Iter2	3.18	3.41	3.29
IRL-Iter1-Policy	6.71	5.96	6.33
IRL-Iter2-Policy	7.01	6.19	6.60

Table 4: Evaluation of Policy Models in MT-Bench. The results show that the IRL method can effectively improve the quality of policy models on multi-turn questions.

Figure 7: Performance comparison of alignment methods. The AIHF-trained model demonstrates superior alignment across various downstream tasks.

Reward Model	Chat	Chat Hard	Safety	Reasoning	Average
DPO Reward Model	37.43%	55.92%	64.14%	47.33%	51.21%
BTL Reward Model	95.11%	56.58%	63.69%	69.22%	71.15%
AIHF Reward Model	94.41%	55.37%	63.98%	76.75%	72.63%

Table 5: Evaluation of reward models in Reward-Bench. It can be seen that the reward model estimated by AIHF which incorporates both demonstrations and preference achieves better performance.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

Reinforcement Learning from Human Feedback: A Statistical Perspective
Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback LLM Alignment
Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.
Unbiased Alignment for Large Language Models with Noisy Preferences
Jul 3, 2026Jialiang Wang, Xianming Liu, Xiong Zhou +2Pairwise Preference Learning LLM Alignment
1Harbin Institute of Technology · 2City University of Hong Kong.
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
May 26, 2026Dongyoon Hahm, Dylan Hadfield-Menell, Kimin LeeLLM Alignment Data Poisoning Attacks
1KAIST · 2MIT

Apr 2, 2026J/K move · Enter open · S save

Reinforcement Learning from Human Feedback: A Statistical Perspective

Pangpang Liu, Chengchun Shi, Will Wei Sun

Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.

RL from Human Feedback LLM Alignment Reward Modeling Preference Learning

Open paper AskPDF arXiv

Reinforcement learning from human feedback (RLHF) has emerged as a central framework for aligning large language models (LLMs) with human preferences. Despite its practical success, RLHF raises fundamental statistical questions because it relies on noisy, subjective, and often heterogeneous feedback to learn reward models and optimize policies. This survey provides a statistical perspective on RLHF, focusing primarily on the LLM alignment setting. We introduce the main components of RLHF, including supervised fine-tuning, reward modeling, and policy optimization, and relate them to familiar statistical ideas such as Bradley-Terry-Luce (BTL) model, latent utility estimation, active learning, experimental design, and uncertainty quantification. We review methods for learning reward functions from pairwise preference data and for optimizing policies through both two-stage RLHF pipelines and emerging one-stage approaches such as direct preference optimization. We further discuss recent extensions including reinforcement learning from AI feedback, inference-time algorithms, and reinforcement learning from verifiable rewards, as well as benchmark datasets, evaluation protocols, and open-source frameworks that support RLHF research. We conclude by highlighting open challenges in RLHF. An accompanying GitHub demo https://github.com/Pangpang-Liu/RLHF_demo illustrates key components of the RLHF pipeline.

Abstract

LLM Alignment

Explore similar work

Reinforcement Learning from Human Feedback: A Statistical Perspective

Unbiased Alignment for Large Language Models with Noisy Preferences

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases