DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment

Liang Zhu; Feiteng Fang; Yuelin Bai; Longze Chen; Zhexiang Zhang; Minghuan Tan; Min Yang

cs.CLApr 2, 2026

DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment

Authors: Liang Zhu, Feiteng Fang, Yuelin Bai, Longze Chen, Zhexiang Zhang, Minghuan Tan, Min Yang

Organizations: Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Science and Technology of China · University of Copenhagen

Abstract

Reinforcement Learning from Human Feedback (RLHF), using algorithms like Proximal Policy Optimization (PPO), aligns Large Language Models (LLMs) with human values but is costly and unstable. Alternatives have been proposed to replace PPO or integrate Supervised Fine-Tuning (SFT) and contrastive learning for direct fine-tuning and value alignment. However, these methods still require voluminous data to learn preferences and may weaken the generalization ability of LLMs. To further enhance alignment efficiency and performance while mitigating the loss of generalization ability, this paper introduces Distribution-guided Efficient Fine-Tuning (DEFT), an efficient alignment framework incorporating data filtering and distributional guidance by calculating the differential distribution reward based on the output distribution of language model and the discrepancy distribution of preference data. A small yet high-quality subset is filtered from the raw data using a differential distribution reward, which is then incorporated into existing alignment methods to guide the model's output distribution. Experimental results demonstrate that the methods enhanced by DEFT outperform the original methods in both alignment capability and generalization ability, with significantly reduced training time.

Figures & tables

Figure 1: The positive and negative distribution can be obtained by calculating word frequencies from the tokenized preference data. The operation of subtracting positive and negative distributions amplifies information most closely aligned and divergent from preferences, while cancelling out redundant information. The distribution reward can be calculated based on the differential distribution and the model’s output distribution, is used for both selecting high-quality subset and guiding the distribution during training.

Figure 2: Data filtration is achieved through pre-computed $R_{Q}$ , where responses demanding preferences of high specificity yield lower $R_{Q}$ , while those unrelated to preferences receive heightened $R_{Q}$ , facilitating the extraction of a dataset characterized by maximal preference information.

Dataset	Method	Harmlessness			Helpfulness			Total
Dataset	Method	BLEU	BART	Reward	BLEU	BART	Reward	BLEU	BART	Reward
0-shot	Llama3-Base	10.51	1.80	53.23	18.74	2.02	46.97	16.51	1.96	48.66
	Llama3-Instruct	23.00	3.06	66.67	33.47	3.74	65.69	30.64	3.54	65.96
	Mistral-Base	8.10	1.73	53.51	14.18	1.87	45.57	12.54	1.83	47.72
	Mistral-Instruct	30.90	3.33	63.50	34.60	3.90	64.80	33.60	3.74	64.45
	ChatGPT	62.68	10.29	73.01	70.79	11.86	75.11	68.60	11.41	74.54

Table 1: Main results. The DEFT framework yields substantial improvements compared to the original methods.

Method	$\mathcal{D}^{3}_{Q}$	$\mathcal{R}_{Q}$	Harmless			Helpful			Total
Method	$\mathcal{D}^{3}_{Q}$	$\mathcal{R}_{Q}$	BELU	BART	Reward	BELU	BART	Reward	BELU	BART	Reward
DEFT-PRO	✓	✓	32.77	3.79	73.79	34.66	3.65	71.24	34.15	3.69	71.93
-	✓		31.76	3.71	73.74	34.51	3.59	70.85	33.77	3.62	71.63
-		✓	29.40	3.56	72.95	33.50	3.64	68.49	33.50	3.62	69.69
DEFT-DPO	✓	✓	32.03	3.95	71.45	36.77	4.16	73.12	35.49	4.10	72.67
-	✓		31.30	3.84	70.88	36.70	4.13	72.98	35.24	4.05	72.41

Table 2: The absence of each component in DEFT will result in a decline in overall performance.

Figure 3: Augmented reference answers enhanced by ChatGPT contribute to a more reasonable calculation of BLEU and BARTScore.

Method	Writing	Roleplay	Reasoning	Math	Coding	Extraction	STEM	Humanity	Turn 1/2	Avg.
SFT	7.23	6.42	4.07	2.85	4.52	6.45	6.07	7.02	5.82/5.33	5.58
PRO	6.55	5.95	4.35	1.87	4.70	5.20	5.33	6.96	5.64/4.89	5.27
w/ DEFT	6.91	5.93	3.25	3.23	4.35	7.15	6.66	6.85	5.77/5.30	5.54
DPO	7.23	6.47	3.70	2.67	4.42	7.23	6.02	6.63	5.64/5.45	5.55
w/ DEFT	8.63	7.98	6.29	4.65	6.47	8.69	8.23	9.22	7.77/7.25	7.53

Table 3: DEFT framework significantly preserves or enhances generalization capability.

Base	Method	Harmless			Helpful			Total
Base	Method	BELU	BART	Reward	BELU	BART	Reward	BELU	BART	Reward
PRO	Likelihood	11.34	1.99	65.32	22.66	2.63	61.71	19.60	2.44	62.69
	Superfiltering	32.61	3.62	71.04	32.87	3.45	69.42	32.80	3.50	69.86
	DEFT	32.77	3.79	73.79	34.66	3.65	71.24	34.15	3.69	71.93
DPO	Likelihood	16.64	2.17	63.94	29.18	2.99	64.42	25.79	2.74	64.29
	Superfiltering	31.92	3.52	69.23	35.36	3.78	70.98	34.43	3.71	70.51

Table 4: The comparison of different data selection methods shows that DEFT outperformed both the high likelihood-based method and Superfiltering across all metrics, further demonstrating its effectiveness.

Figure 4: In both the Harmless and Helpful aspects of human evaluations, the DEFT series demonstrates a higher win rate compared to the original method.

Figure 5: Changes of $\mathcal{R}_{Q}$ during the training process with and without the involvement of $\mathcal{R}_{Q}$ updates (Left). Performance on the test set across varying data volumes (Right).

$\Delta$ Count	[0,5)	[5,20)	[20,50)	50,100)	[100,200)	[200,+ $\infty$ )
Percentage	72.01%	21.82%	3.83%	1.17%	0.61%	0.56%

Table 5: The proportion of tokens with different

\Delta

Count to the total output tokens.

$\Delta$ Frequency	[0,0.001%)	[0.001%,0.005%)	[0.005%,0.01%)	[0.01%,0.02%)	[0.02%,0.05%)	[0.05%,+ $\infty$ )
Percentage	78.57%	17.52%	2.01%	0.93%	0.55%	0.42%

Table 6: The proportion of tokens with different

\Delta

Frequency to the total output tokens.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	Training set				Test
Subset	$\mathcal{D}^{2}$	$\mathcal{D}^{3}$	$\mathcal{D}_{Q}^{2}$	$\mathcal{D}_{Q}^{3}$	Test
$\textup{Harmless}_{\textup{base}}$	42,536		2,127		2,312
$\textup{Helpful}_{\textup{base}}$	43,835		2,192		2,354
$\textup{Helpful}_{\textup{online}}$	22,002		1,101		1,137
$\textup{Helpful}_{\textup{rejection}}$	52,420		2,621		2,749
Total	160,793		8,041		8,552

Appendix

Table 12

Parameter	DEFT-PRO	DEFT-DPO
Epoch	2	2
SFT weight	5e-2	5e-2
Learning rate	5e-6	5e-7
Input length	512	512
Inference length	128	128
$\omega$	1.2e-6	1.2e-7

Appendix

Table 13

Token	$\Delta$ Count	$\Delta$ Frequency
There	487	0.075%
Here	482	0.074%
sorry	476	0.066%
very	465	0.074%
may	414	0.057%
harmful	387	0.060%

Appendix

Table 7: The differences in the statistics of all output tokens on the test set between DEFT and the original method (DEFT - Vanilla).

Token	$\Delta$ Count	$\Delta$ Frequency
nuclear	-6	-0.001%
regardless	-96	-0.017%
anything	-107	-0.019%
access	-129	-0.023%
Yes	-162	-0.036%
Instead	-176	-0.030%

Appendix

Table 8: The differences in the statistics of all output tokens on the test set between DEFT and the original method (DEFT - Vanilla).

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia
Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
Apr 14, 2026Shanyong Wang, Shuhang Lin, Yining Zhao +2LLM Fine-Tuning Personalized Language Model Alignment
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign
Gradient-Guided Reward Optimization for Inference-time Alignment
Jun 8, 2026Hankun Lin, Ruqi ZhangReward Hacking LLM Alignment
Department of Computer Science, Purdue University

Apr 26, 2026J/K move · Enter open · S save

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

Imranul Ashrafi, Inigo Jauregi Unanue, Massimo Piccardi

University of Technology Sydney, Australia

LLM Alignment Representation Engineering Multi-Objective Language Model Alignment Preference Alignment Inference-Time Language Model Alignment Preference Learning

Open paper AskPDF arXiv

Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.

Abstract

LLM Alignment

Explore similar work

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Gradient-Guided Reward Optimization for Inference-time Alignment