Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

Giuseppe Curci; Stefano Simonazzi; Andrea Molinari; Andrea Zugarini

cs.LGOct 27, 2026

Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

Authors: Giuseppe Curci, Stefano Simonazzi, Andrea Molinari, Andrea Zugarini

Organizations: Okkam, Italy · Villanova.ai, Italy · University of Trento, Italy

Abstract

Direct Preference Optimization (DPO) typically relies on preference pairs generated from diverse model pools and ranked by an external LLM judge. The recently proposed delta-learning hypothesis suggests that comparable supervision can instead be obtained from a single pair of models by treating the larger model's response as preferred. We investigate the limits of this approach across model families and scales. We show, both theoretically and empirically, that fixed model pairs introduce systematic, generator-dependent biases that are predictive of the preference label independent of response quality. Consequently, DPO trained on single-pair data consistently underperforms multi-model preference data, despite the larger generator producing higher-quality responses, and we trace this gap to training dynamics consistent with reward hacking on these biases. We further show that simply increasing the number of generators, even without an external judge, is enough to partially mitigate this effect. Our results highlight generator diversity as a critical factor in preference-data construction.

Figures & tables

Panel of Figure 1 — Figure 1: Delta learning underperforms multiple-models. Relative score changes across benchmarks and models using olmo2-1b-prefmix as base dataset. Performance is compared between data generated using multiple models (blue) versus a single pair (orange) ( Qwen3-4B vs Qwen3-0.6B ), with dotted lines indicating average performance.

Panel of Figure 2 — Figure 2: Lower performance doesn’t depend on the generators. Relative score changes across benchmarks. Performance is compared between data generated using multiple models versus a single pair , with dotted lines indicating average performance. Thick orange line represents the average per-benchmark across pairs used.

Model / Dataset	IFEval	TQA	MATH-500	GSM8K	Avg
OLMo2-1B-SFT	61.63%	26.93%	14.60%	51.86%	38.75%
olmo2-1b-prefmix	+7.79%	+4.53%	+0.60%	+6.29%	+4.80%
Average Pairs	-0.31%	+0.34%	-1.12%	+0.50%	-0.14%

Table 1: Delta pair doesn’t underperform because of model-dataset alignment. Benchmark deltas using the original olmo2-1b-prefmix dataset and different model pairs to generate the completions.

Panel of Figure 3 — Figure 3: Training statistics. Mean reward accuracy and margin $\pm$ standard error across models and hyperparameters for olmo2-1b-prefmix and one pair .

Chosen/Dataset	Rejected	Pct Longer%	Median $\Delta$
olmo2-1b-prefmix	-	55.25%	64
Qwen3-4B	Qwen3-0.6B	81.66%	510
Qwen3-32B	Qwen3-0.6B	81.49%	524
Gemma2-27B-it	Gemma2-9B-it	86.96%	723
OLMo2-7B-Instruct	OLMo2-1B-Instruct	47.72%	0.0
OLMo2-32B-Instruct	OLMo2-1B-Instruct	43.84%	-8.0

Table 2: Length Bias. Length completion characters difference between pairs used in Section 3.1 .

Panel of Figure 4 — Figure 4: Style Bias. PCA of model pairs showing trivial separability with Logistic Regression scores and Explained Variance in the top-left.

Panel of Figure 5 — Figure 5: Using more models improves performance. Relative score changes across benchmarks. Performance is compared between data generated using multiple models with judge , average of single pairs and multiple models with delta heuristic , with dotted lines indicating average performance.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Feature Name	Type	Detection Criterion / Pattern
has_boxed_latex	Binary	Contains \boxed or $\boxed
has_hash_delimiter	Binary	Contains #### (GSM8K answer marker)
has_calc_annotations	Binary	Matches calculator annotations <<...>>
has_display_math_dollars	Binary	Contains display math delimiters $$
has_display_math_brackets	Binary	Contains display math delimiters $and$
has_bold_answer	Binary	Matches answer: prefix (case-insensitive)

Appendix

Table 3: GSM8K Style Features. Description and detection criteria used to extract the 11-dimensional style representation vector.

Benchmark	CoT	Shots	Metric	Acronym
TruthfulQA	✗	6	MC1	TQA
Arc-Challenge	✗	0	Unconditional Accuracy	ARC-C
IFEval	✗	0	pass@1 (instance;loose)	-
IFBench	✗	0	pass@1 (instance;loose)	-
Math-500	✓	4	Flex EM	-
GSM8K	✓	8	EM	-

Appendix

Table 4: Overview of evaluation benchmarks. All benchmarks are evaluated using a multi-turn chat format with Chain-of-Thought (CoT) ( Wei et al., 2023 ) and few-shot examples injected as conversation turns where specified. The acronyms are used in the figures.

Model / Dataset	IFEval	TQA	ARC-C	MATH-500	GSM8K	MBPP	H-EVAL	Avg
Qwen2-1.5B-Instruct	$40.17$ %	$28.40$ %	$45.48$ %	$22.80$ %	$53.37$ %	$44.75$ %	$46.34$ %	$40.187\,142\,857\,142\,86$ %
olmo2-1b-prefmix	+ $9.23$ %	+ $1.22$ %	+ $2.68$ %	$0.00$ %	+ $3.64$ %	+ $4.28$ %	+ $1.22$ %	+ $3.181\,428\,571\,428\,571$ %
olmo2-1b-prefmix-delta	$-0.36$ %	+ $0.49$ %	+ $0.67$ %	$-0.60$ %	+ $0.91$ %	+ $1.17$ %	$-1.22$ %	+ $0.151\,428\,571\,428\,571\,4$ %
Gemma2-2B-it	$67.87$ %	$41.62$ %	$48.49$ %	$15.00$ %	$54.51$ %	$50.19$ %	$40.24$ %	$45.417\,142\,857\,142\,86$ %
olmo2-1b-prefmix	+ $12.11$ %	+ $1.59$ %	+ $1.34$ %	+ $1.40$ %	+ $0.91$ %	+ $2.72$ %	+ $6.71$ %	+ $3.825\,714\,285\,714\,286$ %
olmo2-1b-prefmix-delta	+ $4.32$ %	+ $1.22$ %	+ $2.34$ %	$-1.40$ %	+ $2.27$ %	$-1.56$ %	+ $4.27$ %	+ $1.637\,142\,857\,142\,857$ %

Appendix

Table 5: Delta pair generation underperforms. Benchmark deltas using the original olmo2-1b-prefmix dataset and Qwen3-4B and Qwen3-0.6B to generate completions.

Model / Dataset	IFEval	TQA	ARC-C	MATH-500	GSM8K	MBPP	H-EVAL	Avg
Qwen2-1.5B-Instruct	$40.17$ %	$28.40$ %	$45.48$ %	$22.80$ %	$53.37$ %	$44.75$ %	$46.34$ %	$40.187\,142\,857\,142\,86$ %
olmo2-1b-prefmix	+ $9.23$ %	+ $1.22$ %	+ $2.68$ %	$0.00$ %	+ $3.64$ %	+ $4.28$ %	+ $1.22$ %	+ $3.181\,428\,571\,428\,571$ %
Qwen3/ 4 - 0.6 B	$-0.36$ %	$0.49$ %	$0.67$ %	$-0.60$ %	$0.91$ %	$1.17$ %	$-1.22$ %	$0.151\,428\,571\,428\,571\,4$ %
Qwen3/ 32 - 0.6 B	$0.36$ %	$-0.12$ %	$0.00$ %	$0.00$ %	$0.76$ %	$-0.39$ %	$0.61$ %	$0.174\,285\,714\,285\,714\,3$ %
Olmo2-it/ 32 - 1 B	$5.76$ %	$0.12$ %	$1.00$ %	$1.60$ %	$-0.83$ %	$1.56$ %	$0.00$ %	$1.315\,714\,285\,714\,286$ %
Olmo2-it/ 7 - 1 B	$4.32$ %	$-0.24$ %	$2.34$ %	$1.00$ %	$-0.76$ %	$0.00$ %	$-1.22$ %	$0.777\,142\,857\,142\,857\,1$ %

Appendix

Table 6: Ablation pair generators show consistent low performance. Benchmark deltas using the original olmo2-1b-prefmix dataset and different model pairs to generate the completions.

Model / Dataset	IFEval	TQA	MATH-500	GSM8K	Avg
OLMo2-1B-SFT	61.63%	26.93%	14.60%	51.86%	38.75%
olmo2-1b-prefmix	+7.79%	+4.53%	+0.60%	+6.29%	+4.80%
Qwen3/ 4 - 0.6 B	0.24%	1.10%	-0.80%	1.29%	0.46%
Qwen3/ 32 - 0.6 B	1.08%	1.59%	-1.20%	1.21%	0.67%
Olmo2-it/ 32 - 1 B	-0.48%	-0.24%	-0.60%	-0.45%	-0.44%
Olmo2-it/ 7 - 1 B	-1.32%	-0.37%	-1.40%	-0.45%	-0.88%

Appendix

Table 7: Delta pair doesn’t underperform because of model-dataset alignment. Benchmark deltas using the original olmo2-1b-prefmix dataset and different model pairs to generate the completions training OLMo2-1B-SFT , the model for which the dataset was designed.

Model / Dataset	IFEval	TQA	ARC-C	MATH-500	GSM8K	MBPP	H-EVAL	Avg
Qwen2-1.5B-Instruct	$40.17$ %	$28.40$ %	$45.48$ %	$22.80$ %	$53.37$ %	$44.75$ %	$46.34$ %	$40.187\,142\,857\,142\,86$ %
olmo2-1b-prefmix	+ $9.23$ %	+ $1.22$ %	+ $2.68$ %	$0.00$ %	+ $3.64$ %	+ $4.28$ %	+ $1.22$ %	+ $3.181\,428\,571\,428\,571$ %
Average Pairs	+ $2.30$ %	+ $0.02$ %	+ $0.94$ %	+ $0.48$ %	$-0.03$ %	+ $0.62$ %	$-0.49$ %	+ $0.548\,571\,428\,571\,428\,6$ %
Few Pairs	+ $2.04$ %	$-0.37$ %	+ $0.33$ %	+ $1.80$ %	$-1.44$ %	+ $1.95$ %	+ $1.22$ %	+ $0.79$ %
More Pairs	+ $6.36$ %	$-0.24$ %	+ $0.33$ %	+ $2.00$ %	+ $0.76$ %	+ $1.56$ %	$-0.61$ %	+ $1.451\,428\,571\,428\,571$ %
Gemma2-2B-it	$67.87$ %	$41.62$ %	$48.49$ %	$15.00$ %	$54.51$ %	$50.19$ %	$40.24$ %	$45.417\,142\,857\,142\,86$ %

Appendix

Table 8: Using Multiple Pairs. Benchmark deltas using the original olmo2-1b-prefmix dataset and multiple pairs to generate the completions. Few Pairs and More Pairs correspond respectively to completions generated by only the models used in previous experiments and the ones listed in Appendix D.2

Model	GSM8K	IFEval	MBPP	Avg
Gemma2-27B-it	80.67%	77.45%	78.60%	78.91%
Qwen2.5-7B-Instruct	83.70%	71.35%	73.93%	76.33%
Gemma2-9B-it	79.53%	73.01%	65.76%	72.77%
OLMo2-1B-Instruct	67.48%	63.96%	22.57%	51.33%
SmolLM2-1.7B-Instruct	46.25%	46.95%	49.81%	47.67%
Mistral-7B-Instruct-v0.2	45.03%	48.98%	39.30%	44.44%

Appendix

Table 9: Models generator evaluation. Performance across Math, Instruction-Following, and Code benchmarks to determine pairs group for DPO. For IFEval we report prompt-level strict accuracy, as a stricter metric widens the performance gap across models.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 165 papers

Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?
Apr 9, 2026Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni +6Preference Optimization Pairwise Preference Learning
Capital One
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
Oct 6, 2026Yang Qu, Yusheng Han, Chengjia Feng +1Direct Preference Optimization LLM Alignment
Northeastern University

Apr 9, 2026J/K move · Enter open · S save

Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng+5

Capital One

Preference Optimization Pairwise Preference Learning Efficient Language Model Reasoning RL for Language Model Reasoning Data Selection

Open paper AskPDF arXiv

Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples.

Abstract

Direct Preference Optimization

Explore similar work

Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

Boosting Direct Preference Optimization with Penalization

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization