DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment
Authors: Liang Zhu, Feiteng Fang, Yuelin Bai, Longze Chen, Zhexiang Zhang, Minghuan Tan, Min Yang
Organizations: Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Science and Technology of China · University of Copenhagen
Reinforcement Learning from Human Feedback (RLHF), using algorithms like Proximal Policy Optimization (PPO), aligns Large Language Models (LLMs) with human values but is costly and unstable. Alternatives have been proposed to replace PPO or integrate Supervised Fine-Tuning (SFT) and contrastive learning for direct fine-tuning and value alignment. However, these methods still require voluminous data to learn preferences and may weaken the generalization ability of LLMs. To further enhance alignment efficiency and performance while mitigating the loss of generalization ability, this paper introduces Distribution-guided Efficient Fine-Tuning (DEFT), an efficient alignment framework incorporating data filtering and distributional guidance by calculating the differential distribution reward based on the output distribution of language model and the discrepancy distribution of preference data. A small yet high-quality subset is filtered from the raw data using a differential distribution reward, which is then incorporated into existing alignment methods to guide the model's output distribution. Experimental results demonstrate that the methods enhanced by DEFT outperform the original methods in both alignment capability and generalization ability, with significantly reduced training time.
Figures & tables
Figure 1: The positive and negative distribution can be obtained by calculating word frequencies from the tokenized preference data. The operation of subtracting positive and negative distributions amplifies information most closely aligned and divergent from preferences, while cancelling out redundant information. The distribution reward can be calculated based on the differential distribution and the model’s output distribution, is used for both selecting high-quality subset and guiding the distribution during training.
Figure 2: Data filtration is achieved through pre-computed RQ , where responses demanding preferences of high specificity yield lower RQ , while those unrelated to preferences receive heightened RQ , facilitating the extraction of a dataset characterized by maximal preference information.
Dataset
Method
Harmlessness
Helpfulness
Total
BLEU
BART
Reward
BLEU
BART
Reward
BLEU
BART
Reward
0-shot
Llama3-Base
10.51
1.80
53.23
18.74
2.02
46.97
16.51
1.96
48.66
Llama3-Instruct
23.00
3.06
66.67
33.47
3.74
65.69
30.64
3.54
65.96
Mistral-Base
8.10
1.73
53.51
14.18
1.87
45.57
12.54
1.83
47.72
Mistral-Instruct
30.90
3.33
63.50
34.60
3.90
64.80
33.60
3.74
64.45
ChatGPT
62.68
10.29
73.01
70.79
11.86
75.11
68.60
11.41
74.54
Table 1: Main results. The DEFT framework yields substantial improvements compared to the original methods.
Method
DQ3
RQ
Harmless
Helpful
Total
BELU
BART
Reward
BELU
BART
Reward
BELU
BART
Reward
DEFT-PRO
✓
✓
32.77
3.79
73.79
34.66
3.65
71.24
34.15
3.69
71.93
-
✓
31.76
3.71
73.74
34.51
3.59
70.85
33.77
3.62
71.63
-
✓
29.40
3.56
72.95
33.50
3.64
68.49
33.50
3.62
69.69
DEFT-DPO
✓
✓
32.03
3.95
71.45
36.77
4.16
73.12
35.49
4.10
72.67
-
✓
31.30
3.84
70.88
36.70
4.13
72.98
35.24
4.05
72.41
Table 2: The absence of each component in DEFT will result in a decline in overall performance.
Figure 3: Augmented reference answers enhanced by ChatGPT contribute to a more reasonable calculation of BLEU and BARTScore.
Method
Writing
Roleplay
Reasoning
Math
Coding
Extraction
STEM
Humanity
Turn 1/2
Avg.
SFT
7.23
6.42
4.07
2.85
4.52
6.45
6.07
7.02
5.82/5.33
5.58
PRO
6.55
5.95
4.35
1.87
4.70
5.20
5.33
6.96
5.64/4.89
5.27
w/ DEFT
6.91
5.93
3.25
3.23
4.35
7.15
6.66
6.85
5.77/5.30
5.54
DPO
7.23
6.47
3.70
2.67
4.42
7.23
6.02
6.63
5.64/5.45
5.55
w/ DEFT
8.63
7.98
6.29
4.65
6.47
8.69
8.23
9.22
7.77/7.25
7.53
Table 3: DEFT framework significantly preserves or enhances generalization capability.
Base
Method
Harmless
Helpful
Total
BELU
BART
Reward
BELU
BART
Reward
BELU
BART
Reward
PRO
Likelihood
11.34
1.99
65.32
22.66
2.63
61.71
19.60
2.44
62.69
Superfiltering
32.61
3.62
71.04
32.87
3.45
69.42
32.80
3.50
69.86
DEFT
32.77
3.79
73.79
34.66
3.65
71.24
34.15
3.69
71.93
DPO
Likelihood
16.64
2.17
63.94
29.18
2.99
64.42
25.79
2.74
64.29
Superfiltering
31.92
3.52
69.23
35.36
3.78
70.98
34.43
3.71
70.51
Table 4: The comparison of different data selection methods shows that DEFT outperformed both the high likelihood-based method and Superfiltering across all metrics, further demonstrating its effectiveness.
Figure 4: In both the Harmless and Helpful aspects of human evaluations, the DEFT series demonstrates a higher win rate compared to the original method.
Figure 5: Changes of RQ during the training process with and without the involvement of RQ updates (Left). Performance on the test set across varying data volumes (Right).
Δ Count
[0,5)
[5,20)
[20,50)
50,100)
[100,200)
[200,+ ∞ )
Percentage
72.01%
21.82%
3.83%
1.17%
0.61%
0.56%
Table 5: The proportion of tokens with different Δ Count to the total output tokens.
Δ Frequency
[0,0.001%)
[0.001%,0.005%)
[0.005%,0.01%)
[0.01%,0.02%)
[0.02%,0.05%)
[0.05%,+ ∞ )
Percentage
78.57%
17.52%
2.01%
0.93%
0.55%
0.42%
Table 6: The proportion of tokens with different Δ Frequency to the total output tokens.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Subset
Training set
Test
D2
D3
DQ2
DQ3
Harmlessbase
42,536
2,127
2,312
Helpfulbase
43,835
2,192
2,354
Helpfulonline
22,002
1,101
1,137
Helpfulrejection
52,420
2,621
2,749
Total
160,793
8,041
8,552
Appendix
Table 12
Parameter
DEFT-PRO
DEFT-DPO
Epoch
2
2
SFT weight
5e-2
5e-2
Learning rate
5e-6
5e-7
Input length
512
512
Inference length
128
128
ω
1.2e-6
1.2e-7
Appendix
Table 13
Token
Δ Count
Δ Frequency
There
487
0.075%
Here
482
0.074%
sorry
476
0.066%
very
465
0.074%
may
414
0.057%
harmful
387
0.060%
Appendix
Table 7: The differences in the statistics of all output tokens on the test set between DEFT and the original method (DEFT - Vanilla).
Token
Δ Count
Δ Frequency
nuclear
-6
-0.001%
regardless
-96
-0.017%
anything
-107
-0.019%
access
-129
-0.023%
Yes
-162
-0.036%
Instead
-176
-0.030%
Appendix
Table 8: The differences in the statistics of all output tokens on the test set between DEFT and the original method (DEFT - Vanilla).
Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.
Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.
Shanyong Wang, Shuhang Lin, Yining Zhao +2
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign
Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation. While inference-time alignment methods such as Best-of-N and rejection sampling are widely used, they frame the task as a sampling-intensive, reward-guided search, leading to two key limitations: their performance is bounded by the base model's generation quality, and their reliance on imperfect reward models makes them vulnerable to reward hacking. To address these challenges, we introduce Gradient-Guided Reward Optimization (GGRO), a lightweight inference-time method that performs targeted, minimal intervention during decoding via gradient guidance. Specifically, GGRO monitors token-level entropy to identify high-uncertainty regions indicative of drift or misalignment. Upon detection, it responds by injecting nudging tokens, generated using gradient signals from an off-the-shelf reward model, to steer the generation trajectory rather than merely re-ranking samples. Experiments show that GGRO consistently improves inference-time alignment across safety, helpfulness, and reasoning benchmarks. It also increases coverage of high-quality responses and robustness to reward hacking, with minimal computational overhead. Code is available at https://github.com/lhk2004/GGRO.