Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilities. While previous works primarily frame this problem as an optimization or architectural challenge, the inherent characteristics of preference data that drive this degradation remain largely underexplored. In this paper, we propose BALIGN, a balanced data selection strategy that explicitly mitigates catastrophic forgetting while optimizing alignment efficacy. Through theoretical and empirical analyses of the preference optimization gradient, we identify three key data-centric features that dictate parameter drift: the reference model's log-probability margin, the token length difference between chosen and rejected responses, and the TF-IDF similarity to general capability corpora. By aggregating these orthogonal features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples that disrupt intrinsic model parameters or provide minimal alignment utility. Extensive experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains, consistently achieving the optimal Pareto frontier with minimal computational overhead.
Figures & tables
Figure 1: (a) Catastrophic forgetting of general capabilities (e.g., general knowledge, instruction following, reasoning, code, and math) in LLM alignment. (b) Overview of BALIGN, which selects low-risk data for stability and plasticity using risk scores based on three features: reference margin, token length difference, and TF–IDF similarity.
Bucket
sΔpref
sΔℓ
sτ
F ↓
R ↑
F ↓
R ↑
F ↓
R ↑
B1 (lowest risk)
0.08
3.33
0.38
3.20
0.95
3.80
B2
0.37
3.46
0.82
2.95
1.43
3.93
B3
0.52
4.02
1.77
2.26
0.73
3.42
B4
0.84
3.33
1.35
3.20
0.84
2.61
B5 (highest risk)
3.48
1.75
2.19
3.12
0.85
1.67
Table 1: Analysis of catastrophic forgetting F (lower is better) and alignment gain R (higher is better) for each risk score.
Method
Helpfulness
Harmlessness
Llama-3.1-8B-Instruct
Qwen2.5-7B-Instruct
Llama-3.1-8B-Instruct
Qwen2.5-7B-Instruct
General (F ↓ )
Align (R ↑ )
General (F ↓ )
Align (R ↑ )
General (F ↓ )
Align (R ↑ )
General (F ↓ )
Align (R ↑ )
Base
76.86 (–)
61.64 (–)
81.52 (–)
62.84 (–)
76.86 (–)
45.25 (–)
81.52 (–)
44.11 (–)
Full
71.51 (5.35 ↓ )
65.40 (3.76 ↑ )
79.21 (2.31 ↓ )
66.17 (3.33 ↑ )
71.44 (5.42 ↓ )
53.92 (8.67 ↑ )
80.03 (1.49 ↓ )
52.17 (8.06 ↑ )
Random
74.77 (2.09 ↓ )
64.80 (3.16 ↑ )
80.39 (1.13 ↓ )
64.76 (1.92 ↑ )
75.78 (1.08 ↓ )
52.25 (7.00 ↑ )
80.82 (0.70 ↓ )
47.57 (3.46 ↑ )
DSIR
74.89 (1.97 ↓ )
63.44 (1.80 ↑ )
80.30 (1.22 ↓ )
64.50 (1.66 ↑ )
75.10 (1.76 ↓ )
50.77 (5.52 ↑ )
80.86 (0.66 ↓ )
47.83 (3.72 ↑ )
Table 2: General capability and alignment results on the HH-RLHF dataset, evaluated separately for helpfulness and harmlessness. We use Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models as base reference models. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.
Figure 2: Detailed general capabilities and alignment of LLMs. BALIGN successfully preserves the base model’s diverse general capabilities while achieving alignment performance on par with training on the full preference dataset.
Figure 3: Pareto frontier of BALIGN against baselines on the HH-RLHF dataset.
Method
Helpfulness
Harmlessness
Llama-3.1-8B-Instruct
Qwen2.5-7B-Instruct
Llama-3.1-8B-Instruct
Qwen2.5-7B-Instruct
General (F ↓ )
Align (R ↑ )
General (F ↓ )
Align (R ↑ )
General (F ↓ )
Align (R ↑ )
General (F ↓ )
Align (R ↑ )
Base
76.86 (–)
61.64 (–)
81.52 (–)
62.84 (–)
76.86 (–)
45.25 (–)
81.52 (–)
44.11 (–)
W/o sΔpref
75.18 (1.68 ↓ )
65.01 (3.37 ↑ )
80.50 (1.02 ↓ )
65.34 (2.50 ↑ )
74.67 (2.19 ↓ )
53.04 (7.79 ↑ )
79.94 (1.58 ↓ )
48.70 (4.59 ↑ )
W/o sΔℓ
75.59 (1.27 ↓ )
65.95 (4.31 ↑ )
79.98 (1.54 ↓ )
65.56 (2.72 ↑ )
74.27 (2.59 ↓ )
53.34 (8.09 ↑ )
80.19 (1.33 ↓ )
51.07 (6.96 ↑ )
W/o sτ
75.78 (1.08 ↓ )
64.03 (2.39 ↑ )
80.44 (1.08 ↓ )
64.89 (2.05 ↑ )
75.30 (1.56 ↓ )
51.11 (5.86 ↑ )
80.80 (0.72 ↓ )
47.96 (3.85 ↑ )
Table 3: Ablation study of BALIGN on the HH-RLHF dataset, evaluated separately for helpfulness and harmlessness.
Figure 4: Computation time of data selection baselines.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
MMLU
IFEval
ARC-Challenge
HumanEval
GSM8K
General
Forgetting
Alignment
Gain
Base (Llama-3.1-8B-Instruct)
68.78
80.55
83.62
66.46
84.91
76.86
–
61.64
–
Full
66.22
70.68
81.57
62.80
76.27
71.51
5.35
65.40
3.76
Random
68.22
80.20
83.11
59.15
83.17
74.77
2.09
64.80
3.16
DSIR
68.24
78.60
82.94
62.80
81.88
74.89
1.97
63.44
1.80
LESS
68.05
72.99
82.51
60.98
76.35
72.18
4.68
64.14
2.50
TSDS
68.78
79.67
82.34
64.02
83.17
75.60
1.26
64.56
2.92
Appendix
Table 4: Detailed general capability and alignment results for helpfulness on the HH-RLHF dataset, using Llama-3.1-8B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.
Method
MMLU
IFEval
ARC-Challenge
HumanEval
GSM8K
General
Forgetting
Alignment
Gain
Base (Qwen2.5-7B-Instruct)
73.49
76.37
89.51
84.15
84.08
81.52
–
62.84
–
Full
73.02
70.38
89.51
79.27
83.85
79.21
2.31
66.17
3.33
Random
73.37
74.33
89.16
81.10
84.00
80.39
1.13
64.76
1.92
DSIR
72.85
74.00
89.51
78.66
86.50
80.30
1.22
64.50
1.66
LESS
73.47
72.37
89.33
78.66
85.37
79.84
1.68
63.61
0.77
TSDS
73.34
72.55
89.25
83.54
83.40
80.42
1.10
64.55
1.71
Appendix
Table 5: Detailed general capability and alignment results for helpfulness on the HH-RLHF dataset, using Qwen2.5-7B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.
Method
MMLU
IFEval
ARC-Challenge
HumanEval
GSM8K
General
Forgetting
Alignment
Gain
Base (Llama-3.1-8B-Instruct)
68.78
80.55
83.62
66.46
84.91
76.86
–
45.25
–
Full
65.66
74.12
79.86
60.98
76.57
71.44
5.42
53.92
8.67
Random
67.72
79.34
83.19
65.85
82.79
75.78
1.08
52.25
7.00
DSIR
67.52
77.92
83.02
65.24
81.80
75.10
1.76
50.77
5.52
LESS
67.45
74.47
83.62
69.51
79.15
74.84
2.02
49.19
3.94
TSDS
68.03
78.51
82.34
63.41
83.17
75.09
1.77
50.63
5.38
Appendix
Table 6: Detailed general capability and alignment results for harmlessness on the HH-RLHF dataset, using Llama-3.1-8B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.
Method
MMLU
IFEval
ARC-Challenge
HumanEval
GSM8K
General
Forgetting
Alignment
Gain
Base (Qwen2.5-7B-Instruct)
73.49
76.37
89.51
84.15
84.08
81.52
–
44.11
–
Full
73.00
73.53
88.74
84.15
80.74
80.03
1.49
52.17
8.06
Random
73.16
76.70
88.99
82.93
82.34
80.82
0.70
47.57
3.46
DSIR
73.23
76.55
90.02
81.10
83.40
80.86
0.66
47.83
3.72
LESS
73.47
75.51
89.42
81.10
80.14
79.93
1.59
48.62
4.51
TSDS
73.48
76.60
88.99
82.32
82.41
80.76
0.76
46.96
2.85
Appendix
Table 7: Detailed general capability and alignment results for harmlessness on the HH-RLHF dataset, using Qwen2.5-7B-Instruct as the base reference model. The best and second-best results are highlighted in bold and underline, respectively, excluding the base model performance.
Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.
Peng Lai, He Zhu, Zhiwen Ruan +6
Southern University of Science and Technology · Peking University · MSRA +2
The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.
Jialiang Wang, Xianming Liu, Xiong Zhou +2
1Harbin Institute of Technology · 2City University of Hong Kong.
Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.