Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

Minsu Kim; Jianxun Lian; Xing Xie; Steven Euijong Whang

cs.AIAug 25, 2026

Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

Authors: Minsu Kim, Jianxun Lian, Xing Xie, Steven Euijong Whang

Organizations: KAIST · Microsoft Research Asia · Work done during an internship at Microsoft Research Asia

Abstract

Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilities. While previous works primarily frame this problem as an optimization or architectural challenge, the inherent characteristics of preference data that drive this degradation remain largely underexplored. In this paper, we propose BALIGN, a balanced data selection strategy that explicitly mitigates catastrophic forgetting while optimizing alignment efficacy. Through theoretical and empirical analyses of the preference optimization gradient, we identify three key data-centric features that dictate parameter drift: the reference model's log-probability margin, the token length difference between chosen and rejected responses, and the TF-IDF similarity to general capability corpora. By aggregating these orthogonal features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples that disrupt intrinsic model parameters or provide minimal alignment utility. Extensive experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains, consistently achieving the optimal Pareto frontier with minimal computational overhead.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Catastrophic forgetting of general capabilities (e.g., general knowledge, instruction following, reasoning, code, and math) in LLM alignment. (b) Overview of BALIGN, which selects low-risk data for stability and plasticity using risk scores based on three features: reference margin, token length difference, and TF–IDF similarity.

Bucket	$s_{\Delta p_{\mathrm{ref}}}$		$s_{\Delta\ell}$		$s_{\tau}$
	F $\downarrow$	R $\uparrow$	F $\downarrow$	R $\uparrow$	F $\downarrow$	R $\uparrow$
$\mathcal{B}^{1}$ (lowest risk)	0.08	3.33	0.38	3.20	0.95	3.80
$\mathcal{B}^{2}$	0.37	3.46	0.82	2.95	1.43	3.93
$\mathcal{B}^{3}$	0.52	4.02	1.77	2.26	0.73	3.42
$\mathcal{B}^{4}$	0.84	3.33	1.35	3.20	0.84	2.61
$\mathcal{B}^{5}$ (highest risk)	3.48	1.75	2.19	3.12	0.85	1.67

Table 1: Analysis of catastrophic forgetting

F

(lower is better) and alignment gain

R

(higher is better) for each risk score.

Method	Helpfulness				Harmlessness
	Llama-3.1-8B-Instruct		Qwen2.5-7B-Instruct		Llama-3.1-8B-Instruct		Qwen2.5-7B-Instruct
	General (F $\downarrow$ )	Align (R $\uparrow$ )	General (F $\downarrow$ )	Align (R $\uparrow$ )	General (F $\downarrow$ )	Align (R $\uparrow$ )	General (F $\downarrow$ )	Align (R $\uparrow$ )
Base	76.86 (–)	61.64 (–)	81.52 (–)	62.84 (–)	76.86 (–)	45.25 (–)	81.52 (–)	44.11 (–)
Full	71.51 (5.35 $\downarrow$ )	65.40 (3.76 $\uparrow$ )	79.21 (2.31 $\downarrow$ )	66.17 (3.33 $\uparrow$ )	71.44 (5.42 $\downarrow$ )	53.92 (8.67 $\uparrow$ )	80.03 (1.49 $\downarrow$ )	52.17 (8.06 $\uparrow$ )
Random	74.77 (2.09 $\downarrow$ )	64.80 (3.16 $\uparrow$ )	80.39 (1.13 $\downarrow$ )	64.76 (1.92 $\uparrow$ )	75.78 (1.08 $\downarrow$ )	52.25 (7.00 $\uparrow$ )	80.82 (0.70 $\downarrow$ )	47.57 (3.46 $\uparrow$ )
DSIR	74.89 (1.97 $\downarrow$ )	63.44 (1.80 $\uparrow$ )	80.30 (1.22 $\downarrow$ )	64.50 (1.66 $\uparrow$ )	75.10 (1.76 $\downarrow$ )	50.77 (5.52 $\uparrow$ )	80.86 (0.66 $\downarrow$ )	47.83 (3.72 $\uparrow$ )

Table 2: General capability and alignment results on the HH-RLHF dataset, evaluated separately for helpfulness and harmlessness. We use Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models as base reference models. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.

Panel of Figure 2 — Figure 2: Detailed general capabilities and alignment of LLMs. BALIGN successfully preserves the base model’s diverse general capabilities while achieving alignment performance on par with training on the full preference dataset.

Panel of Figure 3 — Figure 3: Pareto frontier of BALIGN against baselines on the HH-RLHF dataset.

Method	Helpfulness				Harmlessness
	Llama-3.1-8B-Instruct		Qwen2.5-7B-Instruct		Llama-3.1-8B-Instruct		Qwen2.5-7B-Instruct
	General (F $\downarrow$ )	Align (R $\uparrow$ )	General (F $\downarrow$ )	Align (R $\uparrow$ )	General (F $\downarrow$ )	Align (R $\uparrow$ )	General (F $\downarrow$ )	Align (R $\uparrow$ )
Base	76.86 (–)	61.64 (–)	81.52 (–)	62.84 (–)	76.86 (–)	45.25 (–)	81.52 (–)	44.11 (–)
W/o $s_{\Delta p_{\mathrm{ref}}}$	75.18 (1.68 $\downarrow$ )	65.01 (3.37 $\uparrow$ )	80.50 (1.02 $\downarrow$ )	65.34 (2.50 $\uparrow$ )	74.67 (2.19 $\downarrow$ )	53.04 (7.79 $\uparrow$ )	79.94 (1.58 $\downarrow$ )	48.70 (4.59 $\uparrow$ )
W/o $s_{\Delta\ell}$	75.59 (1.27 $\downarrow$ )	65.95 (4.31 $\uparrow$ )	79.98 (1.54 $\downarrow$ )	65.56 (2.72 $\uparrow$ )	74.27 (2.59 $\downarrow$ )	53.34 (8.09 $\uparrow$ )	80.19 (1.33 $\downarrow$ )	51.07 (6.96 $\uparrow$ )
W/o $s_{\tau}$	75.78 (1.08 $\downarrow$ )	64.03 (2.39 $\uparrow$ )	80.44 (1.08 $\downarrow$ )	64.89 (2.05 $\uparrow$ )	75.30 (1.56 $\downarrow$ )	51.11 (5.86 $\uparrow$ )	80.80 (0.72 $\downarrow$ )	47.96 (3.85 $\uparrow$ )

Table 3: Ablation study of BALIGN on the HH-RLHF dataset, evaluated separately for helpfulness and harmlessness.

Figure 4: Computation time of data selection baselines.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MMLU	IFEval	ARC-Challenge	HumanEval	GSM8K	General	Forgetting	Alignment	Gain
Base (Llama-3.1-8B-Instruct)	68.78	80.55	83.62	66.46	84.91	76.86	–	61.64	–
Full	66.22	70.68	81.57	62.80	76.27	71.51	5.35	65.40	3.76
Random	68.22	80.20	83.11	59.15	83.17	74.77	2.09	64.80	3.16
DSIR	68.24	78.60	82.94	62.80	81.88	74.89	1.97	63.44	1.80
LESS	68.05	72.99	82.51	60.98	76.35	72.18	4.68	64.14	2.50
TSDS	68.78	79.67	82.34	64.02	83.17	75.60	1.26	64.56	2.92

Appendix

Table 4: Detailed general capability and alignment results for helpfulness on the HH-RLHF dataset, using Llama-3.1-8B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.

Method	MMLU	IFEval	ARC-Challenge	HumanEval	GSM8K	General	Forgetting	Alignment	Gain
Base (Qwen2.5-7B-Instruct)	73.49	76.37	89.51	84.15	84.08	81.52	–	62.84	–
Full	73.02	70.38	89.51	79.27	83.85	79.21	2.31	66.17	3.33
Random	73.37	74.33	89.16	81.10	84.00	80.39	1.13	64.76	1.92
DSIR	72.85	74.00	89.51	78.66	86.50	80.30	1.22	64.50	1.66
LESS	73.47	72.37	89.33	78.66	85.37	79.84	1.68	63.61	0.77
TSDS	73.34	72.55	89.25	83.54	83.40	80.42	1.10	64.55	1.71

Appendix

Table 5: Detailed general capability and alignment results for helpfulness on the HH-RLHF dataset, using Qwen2.5-7B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.

Method	MMLU	IFEval	ARC-Challenge	HumanEval	GSM8K	General	Forgetting	Alignment	Gain
Base (Llama-3.1-8B-Instruct)	68.78	80.55	83.62	66.46	84.91	76.86	–	45.25	–
Full	65.66	74.12	79.86	60.98	76.57	71.44	5.42	53.92	8.67
Random	67.72	79.34	83.19	65.85	82.79	75.78	1.08	52.25	7.00
DSIR	67.52	77.92	83.02	65.24	81.80	75.10	1.76	50.77	5.52
LESS	67.45	74.47	83.62	69.51	79.15	74.84	2.02	49.19	3.94
TSDS	68.03	78.51	82.34	63.41	83.17	75.09	1.77	50.63	5.38

Appendix

Table 6: Detailed general capability and alignment results for harmlessness on the HH-RLHF dataset, using Llama-3.1-8B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.

Method	MMLU	IFEval	ARC-Challenge	HumanEval	GSM8K	General	Forgetting	Alignment	Gain
Base (Qwen2.5-7B-Instruct)	73.49	76.37	89.51	84.15	84.08	81.52	–	44.11	–
Full	73.00	73.53	88.74	84.15	80.74	80.03	1.49	52.17	8.06
Random	73.16	76.70	88.99	82.93	82.34	80.82	0.70	47.57	3.46
DSIR	73.23	76.55	90.02	81.10	83.40	80.86	0.66	47.83	3.72
LESS	73.47	75.51	89.42	81.10	80.14	79.93	1.59	48.62	4.51
TSDS	73.48	76.60	88.99	82.32	82.41	80.76	0.76	46.96	2.85

Appendix

Table 7: Detailed general capability and alignment results for harmlessness on the HH-RLHF dataset, using Qwen2.5-7B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 600 papers

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
Sep 5, 2026Peng Lai, He Zhu, Zhiwen Ruan +6Data Selection Preference Alignment
Southern University of Science and Technology · Peking University · MSRA +2
Unbiased Alignment for Large Language Models with Noisy Preferences
Jul 3, 2026Jialiang Wang, Xianming Liu, Xiong Zhou +2Pairwise Preference Learning LLM Alignment
1Harbin Institute of Technology · 2City University of Hong Kong.
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia

Sep 5, 2026J/K move · Enter open · S save

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

Peng Lai, He Zhu, Zhiwen Ruan, Dongdong Zhang+5

Southern University of Science and Technology · Peking University · MSRA+2

Data Selection Preference Alignment

Open paper AskPDF arXiv

Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.

Abstract

LLM Alignment

Explore similar work

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

Unbiased Alignment for Large Language Models with Noisy Preferences

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing