Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Wenbo Zhang; Wenzhuo Zhou; Hengrui Cai; Zhengling Qi

cs.CLSep 7, 2026

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Authors: Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi

Organizations: Department of Statistics University of California, Irvine · School of Business George Washington University

Abstract

Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about $42\%$ , demonstrating both its effectiveness and efficiency.

Figures & tables

Figure 1: Intuition behind our method : Both the explicit preference model $\hat{p}$ and the implicit model $\hat{\pi}_{\text{dpo}}$ aim to approximate the ground-truth preference model $p^{*}$ . For the implicit class, searching for $p^{*}$ is equivalent to searching for the optimal policy $\pi^{*}$ . In practice, we observe that learning $p^{*}$ is easier within the explicit preference class $\mathcal{P}$ than within the implicit policy class $\Pi$ . As a result, $\hat{\pi}_{\text{DP3O}}$ leverages the fitted explicit model as a form of distillation to improve policy alignment.

Panel of Figure 2 — Figure 2: (a) Performance comparison between offline and iterative DPO methods, measured by win rate compared with SFT. Iterative DPO consistently outperforms its offline variant. (b) The distribution of preference probabilities annotated by the implicit and explicit preference model. Explicit preference probability is closer to the true probability (c) Classification accuracy of preference models on generated samples shows that explicit models outperform the implicit model.

Type	Method	Win Rate % ( $\uparrow$ )			$\Delta r$ ( $\uparrow$ )	Training Time (h) ( $\downarrow$ )
Type	Method	Base	Chosen	Average	$\Delta r$ ( $\uparrow$ )	Training Time (h) ( $\downarrow$ )
Offline	DPO	$78.62$	$47.90$	$63.26$	$1.22$	$0.92$
	CDPO	$71.18$	$38.16$	$54.67$	$0.87$	$0.91$
	RDPO	$73.52$	$43.07$	$58.30$	$1.05$	$0.91$
	d-DPO	$79.51$	$49.66$	$64.59$	$1.33$	$1.54$
	DP3O	$\mathbf{82.65}$	$\mathbf{53.24}$	$\mathbf{67.95}$	$\mathbf{1.39}$	$1.55$

Table 1: Controlled experiments: comparison of DP3O and baseline methods in terms of win rate, reward gain

\Delta r

relative to SFT, and training time. Bold indicates the best results among offline methods.

Panel of Figure 3 — Figure 3: Ablation on $\alpha$ and teacher model type. Left : Average win rate of DP3O under different values of $\alpha$ , using the explicit preference model as the default teacher. Right : Comparison of DP3O with two types of teacher models, evaluated by average win rate and reward gain $\Delta r$ .

Type	Method	$\epsilon=0.3$				$\epsilon=0.45$
		Win Rate % ( $\uparrow$ )			$\Delta r$ ( $\uparrow$ )	Win Rate % ( $\uparrow$ )			$\Delta r$ ( $\uparrow$ )
		Base	Chosen	Average		Base	Chosen	Average
Offline	DPO	$65.96$	$32.30$	$49.13$	$0.66$	$52.10$	$23.13$	$37.62$	$0.14$
	CDPO	$58.26$	$26.31$	$42.29$	$0.35$	$51.90$	$23.16$	$37.53$	$0.11$
	RDPO	$71.30$	$38.71$	$55.01$	$0.89$	$51.67$	$22.42$	$37.05$	$0.11$
	d-DPO	$68.37$	$42.14$	$55.26$	$0.91$	$52.39$	$23.27$	$37.83$	$0.16$

Table 2: Results on noisy preference data with label-flipping noise ratios of

0.3

and

0.45

. Bold indicates the best results among all methods.

Method	Mistral-7B			Llama-3-8B
	AlpacaEval 2		MT-Bench	AlpacaEval 2		MT-Bench
	LC Win Rate (%)	Win Rate (%)	GPT-4 Turbo	LC Win Rate (%)	Win Rate (%)	GPT-4 Turbo
SFT	8.4	6.2	4.8	6.2	4.6	5.2
SLiC-HF	10.9	8.9	5.8	12.3	13.7	6.3
DPO	15.1	12.5	$\mathbf{5.9}$	18.2	$\mathbf{15.5}$	$\mathbf{6.5}$
IPO	11.8	9.4	5.5	14.4	14.2	$\mathbf{6.5}$

Table 3: AlpacaEval 2 ( Li et al., 2023 ) and MT-Bench ( Zheng et al., 2023a ) results of two models. Bold indicates the best results among all methods.

Model	Method	MMLU	ARC	HellaSwag	TruthfulQA	Winograd	GSM8K	Average
Mistral 7b	SFT	$60.10$	$58.28$	$80.76$	$40.35$	$76.40$	$28.13$	$57.34$
	SLiC-HF	$59.24$	$55.38$	$81.15$	$48.36$	$77.35$	$33.74$	$59.20$
	DPO	$58.48$	$61.26$	$83.59$	$53.06$	$76.80$	$21.76$	$59.16$
	IPO	$60.23$	$60.84$	$83.30$	$45.44$	$77.58$	$27.14$	$59.09$
	CPO	$59.39$	$57.00$	$80.75$	$47.07$	$76.48$	$33.06$	$58.96$
	d-DPO	$58.14$	$59.22$	$83.71$	$49.23$	$76.13$	$24.24$	$58.45$

Table 4: Results on the open LLM leaderboard. Bold indicates the best results among all methods.

Figure 4: Performance of DP3O, DPO, and iterative DPO on Mistral-7B.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The overall framework for controlled experiments comparing iterative and offline approaches.

Figure 6: Accuracy of implicit models.

Method	ACC (%) ( $\uparrow$ )	ECE ( $\downarrow$ )	Win rate ( $\uparrow$ )
Explicit	$\mathbf{82.4}$	$\mathbf{0.24}$	$\mathbf{77.1}$
LLM-judge (prompt)	$52.2$	$0.46$	$70.4$
LLM-judge (trained)	$68.3$	$0.32$	$74.8$

Appendix

Table 5: Comparison of the explicit preference model and LLM-as-judge baselines.

Figure 7: Randomization test on Pythia-1.4B with randomly assigned training preference labels. Left: Train loss on the randomized-label training set. Explicit (blue, all $\beta$ ) plateaus near the random-guessing loss and does not fit the randomly assigned labels. Implicit DPO (red, all $\beta$ ) drives the train loss close to zero. Right: Eval loss on the clean validation set with original labels. Explicit stays near the random-guessing loss, while the eval loss of Implicit DPO increases, showing that it fits the randomized training labels rather than learning transferable preference structure. The gap is consistent across $\beta$ values, isolating the contribution of the output parameterization from the shared backbone.

Figure 8: Ablation on preference signals in iterative alignment. Iterative DPO with an explicit preference model outperforms its implicit counterpart, while iterative DP3O further improves performance by using explicit soft probabilities.

Input $\to$	Q: How can you determine if a person is genuinely interested in a conversation or simply being polite?
Mistral DPO $\to$	As an AI, I don’t have access to contextual information or nonverbal cues, but here are some general signs that may indicate whether someone is genuinely interested in a conversation or being polite:
	1. Active listening: A genuinely interested person will actively listen to what you’re saying, ask questions, and provide feedback or comments. They may also maintain eye contact, nod, or use verbal cues like “uh-huh” or “I see” to show they’re engaged.
	2. Follow-up questions: A genuine interest in the conversation often leads to follow-up questions. If someone is asking questions that show they’re interested in learning more about the topic, it’s a good sign they’re engaged.
	3. Body language: Nonverbal cues like leaning in, smiling, and mirroring your body language can indicate genuine interest. However, note that some people may have different body language styles, so this may not always be a reliable indicator.
	4. Topic continuation: If the conversation flows naturally, and both parties are contributing equally, it’s a sign of mutual interest. If one person dominates the conversation or keeps steering it back to their own interests, it may indicate a lack of genuine interest.
	5. Time spent: If the conversation lasts longer than expected or planned, it’s a sign that both parties are engaged and interested.

Appendix

Figure 9: Two AlpacaEval 2 generations from the Mistral model after training with DPO and DP3O.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference Learning LLM Alignment
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
Intrinsic Mutual Information as a Modulator for Preference Optimization
Apr 27, 2026Peng Liao, Peijia Zheng, Lingbo Li +2LLM Alignment Offline Preference Optimization
1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.

Sep 16, 2026J/K move · Enter open · S save

A Zeroth-Order Paradigm for LLM Preference Alignment

Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA+1

Pairwise Preference Learning LLM Alignment Preference Optimization

Open paper AskPDF arXiv

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.

Abstract

LLM Alignment

Explore similar work

A Zeroth-Order Paradigm for LLM Preference Alignment

Intrinsic Mutual Information as a Modulator for Preference Optimization

Boosting Direct Preference Optimization with Penalization