Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Wenbo Zhang; Wenzhuo Zhou; Hengrui Cai; Zhengling Qi

cs.CLSep 7, 2026

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Authors: Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi

Organizations: Department of Statistics University of California, Irvine · School of Business George Washington University

Abstract

Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about $42\%$ , demonstrating both its effectiveness and efficiency.

Figures & tables

Figure 1: Intuition behind our method : Both the explicit preference model $\hat{p}$ and the implicit model $\hat{\pi}_{\text{dpo}}$ aim to approximate the ground-truth preference model $p^{*}$ . For the implicit class, searching for $p^{*}$ is equivalent to searching for the optimal policy $\pi^{*}$ . In practice, we observe that learning $p^{*}$ is easier within the explicit preference class $\mathcal{P}$ than within the implicit policy class $\Pi$ . As a result, $\hat{\pi}_{\text{DP3O}}$ leverages the fitted explicit model as a form of distillation to improve policy alignment.

Panel of Figure 2 — Figure 2: (a) Performance comparison between offline and iterative DPO methods, measured by win rate compared with SFT. Iterative DPO consistently outperforms its offline variant. (b) The distribution of preference probabilities annotated by the implicit and explicit preference model. Explicit preference probability is closer to the true probability (c) Classification accuracy of preference models on generated samples shows that explicit models outperform the implicit model.

Type	Method	Win Rate % ( $\uparrow$ )			$\Delta r$ ( $\uparrow$ )	Training Time (h) ( $\downarrow$ )
Type	Method	Base	Chosen	Average	$\Delta r$ ( $\uparrow$ )	Training Time (h) ( $\downarrow$ )
Offline	DPO	$78.62$	$47.90$	$63.26$	$1.22$	$0.92$
	CDPO	$71.18$	$38.16$	$54.67$	$0.87$	$0.91$
	RDPO	$73.52$	$43.07$	$58.30$	$1.05$	$0.91$
	d-DPO	$79.51$	$49.66$	$64.59$	$1.33$	$1.54$
	DP3O	$\mathbf{82.65}$	$\mathbf{53.24}$	$\mathbf{67.95}$	$\mathbf{1.39}$	$1.55$

Table 1: Controlled experiments: comparison of DP3O and baseline methods in terms of win rate, reward gain

\Delta r

relative to SFT, and training time. Bold indicates the best results among offline methods.

Panel of Figure 3 — Figure 3: Ablation on $\alpha$ and teacher model type. Left : Average win rate of DP3O under different values of $\alpha$ , using the explicit preference model as the default teacher. Right : Comparison of DP3O with two types of teacher models, evaluated by average win rate and reward gain $\Delta r$ .

Type	Method	$\epsilon=0.3$				$\epsilon=0.45$
		Win Rate % ( $\uparrow$ )			$\Delta r$ ( $\uparrow$ )	Win Rate % ( $\uparrow$ )			$\Delta r$ ( $\uparrow$ )
		Base	Chosen	Average		Base	Chosen	Average
Offline	DPO	$65.96$	$32.30$	$49.13$	$0.66$	$52.10$	$23.13$	$37.62$	$0.14$
	CDPO	$58.26$	$26.31$	$42.29$	$0.35$	$51.90$	$23.16$	$37.53$	$0.11$
	RDPO	$71.30$	$38.71$	$55.01$	$0.89$	$51.67$	$22.42$	$37.05$	$0.11$
	d-DPO	$68.37$	$42.14$	$55.26$	$0.91$	$52.39$	$23.27$	$37.83$	$0.16$

Table 2: Results on noisy preference data with label-flipping noise ratios of

0.3

and

0.45

. Bold indicates the best results among all methods.

Method	Mistral-7B			Llama-3-8B
	AlpacaEval 2		MT-Bench	AlpacaEval 2		MT-Bench
	LC Win Rate (%)	Win Rate (%)	GPT-4 Turbo	LC Win Rate (%)	Win Rate (%)	GPT-4 Turbo
SFT	8.4	6.2	4.8	6.2	4.6	5.2
SLiC-HF	10.9	8.9	5.8	12.3	13.7	6.3
DPO	15.1	12.5	$\mathbf{5.9}$	18.2	$\mathbf{15.5}$	$\mathbf{6.5}$
IPO	11.8	9.4	5.5	14.4	14.2	$\mathbf{6.5}$

Table 3: AlpacaEval 2 ( Li et al., 2023 ) and MT-Bench ( Zheng et al., 2023a ) results of two models. Bold indicates the best results among all methods.

Model	Method	MMLU	ARC	HellaSwag	TruthfulQA	Winograd	GSM8K	Average
Mistral 7b	SFT	$60.10$	$58.28$	$80.76$	$40.35$	$76.40$	$28.13$	$57.34$
	SLiC-HF	$59.24$	$55.38$	$81.15$	$48.36$	$77.35$	$33.74$	$59.20$
	DPO	$58.48$	$61.26$	$83.59$	$53.06$	$76.80$	$21.76$	$59.16$
	IPO	$60.23$	$60.84$	$83.30$	$45.44$	$77.58$	$27.14$	$59.09$
	CPO	$59.39$	$57.00$	$80.75$	$47.07$	$76.48$	$33.06$	$58.96$
	d-DPO	$58.14$	$59.22$	$83.71$	$49.23$	$76.13$	$24.24$	$58.45$

Table 4: Results on the open LLM leaderboard. Bold indicates the best results among all methods.

Figure 4: Performance of DP3O, DPO, and iterative DPO on Mistral-7B.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The overall framework for controlled experiments comparing iterative and offline approaches.

Figure 6: Accuracy of implicit models.

Method	ACC (%) ( $\uparrow$ )	ECE ( $\downarrow$ )	Win rate ( $\uparrow$ )
Explicit	$\mathbf{82.4}$	$\mathbf{0.24}$	$\mathbf{77.1}$
LLM-judge (prompt)	$52.2$	$0.46$	$70.4$
LLM-judge (trained)	$68.3$	$0.32$	$74.8$

Appendix

Table 5: Comparison of the explicit preference model and LLM-as-judge baselines.

Figure 7: Randomization test on Pythia-1.4B with randomly assigned training preference labels. Left: Train loss on the randomized-label training set. Explicit (blue, all $\beta$ ) plateaus near the random-guessing loss and does not fit the randomly assigned labels. Implicit DPO (red, all $\beta$ ) drives the train loss close to zero. Right: Eval loss on the clean validation set with original labels. Explicit stays near the random-guessing loss, while the eval loss of Implicit DPO increases, showing that it fits the randomized training labels rather than learning transferable preference structure. The gap is consistent across $\beta$ values, isolating the contribution of the output parameterization from the shared backbone.

Figure 8: Ablation on preference signals in iterative alignment. Iterative DPO with an explicit preference model outperforms its implicit counterpart, while iterative DP3O further improves performance by using explicit soft probabilities.

Input $\to$	Q: How can you determine if a person is genuinely interested in a conversation or simply being polite?
Mistral DPO $\to$	As an AI, I don’t have access to contextual information or nonverbal cues, but here are some general signs that may indicate whether someone is genuinely interested in a conversation or being polite:
	1. Active listening: A genuinely interested person will actively listen to what you’re saying, ask questions, and provide feedback or comments. They may also maintain eye contact, nod, or use verbal cues like “uh-huh” or “I see” to show they’re engaged.
	2. Follow-up questions: A genuine interest in the conversation often leads to follow-up questions. If someone is asking questions that show they’re interested in learning more about the topic, it’s a good sign they’re engaged.
	3. Body language: Nonverbal cues like leaning in, smiling, and mirroring your body language can indicate genuine interest. However, note that some people may have different body language styles, so this may not always be a reliable indicator.
	4. Topic continuation: If the conversation flows naturally, and both parties are contributing equally, it’s a sign of mutual interest. If one person dominates the conversation or keeps steering it back to their own interests, it may indicate a lack of genuine interest.
	5. Time spent: If the conversation lasts longer than expected or planned, it’s a sign that both parties are engaged and interested.

Appendix

Figure 9: Two AlpacaEval 2 generations from the Mistral model after training with DPO and DP3O.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

Sep 16, 2026

A Zeroth-Order Paradigm for LLM Preference Alignment

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.

Peter Chen, Xi Chen, Wotao Yin +1

Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1

Pairwise Preference Learning LLM Alignment Preference Optimization

Ask PDF

Apr 27, 2026

Intrinsic Mutual Information as a Modulator for Preference Optimization

Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at https://github.com/liavonpenn/rmipo.

Peng Liao, Peijia Zheng, Lingbo Li +2

1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University

LLM Alignment Offline Preference Optimization LLM Training

Ask PDF

Jun 10, 2026

Boosting Direct Preference Optimization with Penalization

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Pengwei Sun

Stanford University.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Offline Preference Optimization

Ask PDF

Abstract

LLM Alignment

Explore similar work

A Zeroth-Order Paradigm for LLM Preference Alignment

Intrinsic Mutual Information as a Modulator for Preference Optimization

Boosting Direct Preference Optimization with Penalization