AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
Authors: Peng Lai, He Zhu, Zhiwen Ruan, Dongdong Zhang, Yun Chen, Peng Li, Furu Wei, Yang Liu, Guanhua Chen
Organizations: Southern University of Science and Technology · Peking University · MSRA · Shanghai University of Finance and Economics · Tsinghua University
Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.
Figures & tables
Figure 1: Complementarity between inverse and positive preference signals. We present the scatter plot of the joint distribution of positive and inverse implicit reward margins (IM) computed by SFT models on UltraFeedback samples. The results show that inverse IM and positive IM do not exhibit a clear inverse correlation, indicating that inverse signals can provide additional information not captured by positive signals, helping to understand the model’s preference distribution better.
Figure 2: Overview of the AlignDiff Framework . Stage 1 uses Alignment Discrepancy ( RAD ) with positive and inverse implicit reward margins to retain polarity-clear pairs; Stage 2 ranks them by the Average Negative Log-Likelihood Gap (ANG) to prioritize informative hard pairs and mitigate the squeezing effect. AlignDiff relies solely on internal model signals.
Method
LLaMA-3-8B-SFT
Qwen2.5-7B-SFT
Alpaca Eval 2.0
Arena-Hard
MT-Bench
Alpaca Eval 2.0
Arena-Hard
MT-Bench
LC (%)
WR (%)
Len
WR (%)
Score
LC (%)
WR (%)
Len
WR (%)
Score
Init
1.2
2.2
3,841
7.7
5.2 ∗
5.3
4.7
1,230
10.7
5.7
Full
13.7
16.8
3,431
36.2
6.5 ∗
21.3
18.9
1,715
46.9
6.8
PPLGAP
3.7
6.3
4,002
33.7
6.5
21.2
16.9
1,631
39.0
6.7
LCPP
14.6
20.1
5,138
25.0
7.1
20.0
22.6
2,050
38.8
7.2
Table 1: Performance comparison on AlpacaEval 2.0, Arena-Hard, and MT-Bench using DPO-trained models with various data subsets . The base models are LLaMA-3-8B-SFT and Qwen2.5-7B-SFT. The best-performing subsets are highlighted in bold , and the second-best ones are underlined .
Figure 3: Results of Hyperparameter & Ablation Studies . Panels (a) and (b) show the results of our hyperparameter studies; Panel (c) presents the ablation results for the role of each filtering step.
Figure 4: Training Dynamics and Performance . Dynamics of three metrics during DPO training: (a) training loss; (b) rewards of chosen responses; (c) rewards of rejected responses, each showing results for different filtering methods: None, RAD -only, ANG-only, and the full framework; and (d) performance comparison. Although RAD -only achieves lower loss, it is constrained by the squeezing effect; ANG-only mitigates squeezing but struggles to converge without polarity filtering.
Figure 5: (a) RAD -based filtering consistently outperforms PIM, IIM, and EM; (b) calibrating on hard samples boosts performance vs. easy/random baselines.
Figure 6: Dist. of Avg. NLLs. The model trained on filtered data exhibits a tighter, left-shifted distribution.
Sample Ordering Strategy
LC(%)
WR(%)
Length
Random Order (baseline)
33.38
33.81
2,035
Hard-to-Easy
27.72
27.95
2,005
Easy-to-Hard
36.40
36.89
2,064
Table 2: Sample Ordering Strategies Comparison.
Method
GPU Hours
Avg. WR
Efficiency
IM
43
24.6
0.57
EM
128
17.2
0.13
SDPO
162
25.6
0.16
AlignDiff
80.5
30.2
0.38
AlignDiff (Random-Half)
48.5
28.1
0.58
Table 3: Efficiency Comparison across Methods.
Figure 7: External Reward margin optimization with length gap preserved ((a) and (b)). The filtered data caused almost no change in the length gap distribution, while optimizing the external reward margin distribution, reducing ambiguous samples, and shifting the overall distribution slightly to the right. Panel (c) shows the comparative analysis from method and model Perspectives.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
LLaMA-3-8B-SFT
Qwen2.5-7B-SFT
AlpacaEval 2.0
Arena-Hard
AlpacaEval 2.0
Arena-Hard
LC (%)
WR (%)
Len
WR (%)
LC (%)
WR (%)
Len
WR (%)
Init
5.6
10.4
3,841
12.3
8.7
7.5
1,230
13.0
Full
20.9
28.0
3,431
31.5
31.1
26.9
1,715
40.0
PPLGAP
9.8
20.7
4,002
26.6
27.3
22.2
1,631
33.0
LCPP
22.1
33.8
5,138
27.9
29.5
30.8
2,050
34.0
Appendix
Table 4: Average performance comparison of DPO-trained models on AlpacaEval 2.0 and Arena-Hard using different data subsets, evaluated with multiple judge models. . The base models are LLaMA-3-8B-SFT and Qwen2.5-7B-SFT. The best-performing results are highlighted in bold , and the second-best ones are underlined .
Method
GPQA
Toxigen
TruthfulQA
MMLU
Winogrande
AVG
LLaMA-3-8B-SFT
PPLGAP
0.328
0.427
0.493
0.613
0.729
0.518
EM
0.313
0.430
0.521
0.624
0.719
0.521
IM
0.317
0.429
0.488
0.622
0.719
0.515
IM & Em
0.308
0.428
0.490
0.623
0.717
0.513
LCPP
0.332
0.427
0.498
0.612
0.730
0.520
Appendix
Table 5: Performance Comparison of downstream tasks (GPQA / Toxigen / TruthfulQA / MMLU / Winogrande) Using DPO-Trained Models with Various Data Subsets .
Figure 8: Pseudocode of AlignDiff.
Figure 9: Distribution of RAD for three SFT models on the UltraFeedback dataset.
Annotator
Human Agreement
Price
Spearman Corr.
Pearson Corr.
Bias
Variance
DeepSeek-V3
67.27
0.12
0.95
0.87
32.19
16.45
GPT-4 Turbo
65.73
4.32
0.78
0.77
33.90
23.65
Appendix
Table 6: Comparison of AlpacaEval 2.0 Annotators. The better results under each metric in the table are highlighted in bold .
Figure 10: Prompt Template for Scoring Preference Pairs Using Qwen2.5-72B-Instruct .
Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.