On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

Kensuke Sasaki; Issei Sato

cs.LGOct 27, 2026

On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

Authors: Kensuke Sasaki, Issei Sato

Organizations: Department of Computer Science, The University of Tokyo

Abstract

Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement---a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7--9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to $35{\times}$ compared to supervised fine-tuning.

Figures & tables

Figure 1: Distribution of

A_{i}

across data pairs for GPT-2 Small (top) and Medium (bottom) at 1 epoch (left), 100 epochs (middle), and 1000 epochs (right) on HH-RLHF. Dark bars indicate the top 10% of pairs by weight. As training progresses, the distribution polarizes onto a small number of support pairs; this effect is more pronounced in the larger model.

Model	Dataset	Top-10% mass (%)
		Epoch 1	Epoch 100	Epoch 1000
GPT-2 Small	HH-RLHF	$10.88$	$28.78$	$39.57$
GPT-2 Medium	HH-RLHF	$11.02$	$32.32$	$43.70$
		Epoch 1	Epoch 2	Epoch 3
Llama-3-8B-Instruct	UltraFeedback	$11.75$	$41.82$	$52.48$
Mistral-7B-Instruct-v0.2	UltraFeedback	$11.23$	$39.92$	$43.77$

Table 1: Top-10% mass of

A_{i}

N=1000

pairs, i.e., the fraction of

\sum_{i}A_{i}

held by the top 10% of pairs (10% under a uniform distribution). Top: GPT-2 trained with DPO on HH-RLHF. Bottom: 7–8B models trained with full fine-tuning DPO on UltraFeedback Binarized.

Figure 2: Element-wise distribution of gradient feature vectors on a logarithmic scale for GPT-2 Medium on

N=100

pairs from HH-RLHF. (a) DPO: the vast majority of entries are near zero, confirming assumption 2 . (b) SFT: the distribution is substantially denser, since no gradient cancellation occurs. Near-zero fractions for all five models, computed in a separate setting, are reported in table 2 .

Model	Params	DPO $\Phi_{i}$ (%)	SFT $\Phi_{i}^{\mathrm{SFT}}$ (%)
GPT-2 Small	124M	$91.29$	$3.86$
GPT-2 Medium	355M	$94.82$	$4.34$
Llama-3-8B-Instruct	8B	$92.51$	$16.44$
Mistral-7B-Instruct-v0.2	7B	$90.93$	$19.72$
Gemma-2-9B-it	9B	$92.12$	$13.57$

Table 2: Fraction of near-zero entries in a representative gradient feature vector under DPO (

\Phi_{i}

) and SFT (

\Phi_{i}^{\mathrm{SFT}}

), computed from 1,000 pairs sampled from UltraFeedback Binarized. Higher values indicate sparser features.

Task Pair	Method	Model	Jaccard (%)	$\boldsymbol{\langle u_{1}^{\star},V_{2}\rangle}$
HH help $\leftrightarrow$ harm	SFT	Small	$54.9$	$9.12$
	DPO	Small	$18.9$	$0.258$
	SFT	Medium	$49.3$	$3.22$
	DPO	Medium	$11.4$	$0.191$
HH help $\leftrightarrow$ UF	SFT	Small	$68.4$	$21.2$
	DPO	Small	$25.2$	$1.91$

Table 3: Cross-task interference for DPO and SFT across two model scales (GPT-2 Small/Medium) and two task pairs: HH help

\leftrightarrow

harm (HH-RLHF helpful vs. harmless subsets) and HH help

\leftrightarrow

UF (HH-RLHF helpful vs. UltraFeedback). Lower values indicate greater disentanglement. Across all configurations, DPO reduces the interference

\langle u_{1}^{\star},V_{2}\rangle

11\times

35\times

relative to SFT.

Task Pair	Method	$\Delta$ Task 1	$\Delta$ Task 2
HH help $\leftrightarrow$ harm	SFT	$0.110$	$0.102$
HH help $\leftrightarrow$ harm	DPO	$\mathbf{0.027}$	$\mathbf{0.026}$
HH help $\leftrightarrow$ UF	SFT	$0.096$	$0.030$
HH help $\leftrightarrow$ UF	DPO	$\mathbf{0.036}$	$\mathbf{0.014}$

Table 4: Performance degradation

\Delta

(single-task

-

merged; lower is better) under task-vector merging on GPT-2 Medium. Task 1 is HH helpful; Task 2 is HH harmless or UltraFeedback. Full results are reported in appendix D .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\lambda_{\max}(K)/\tilde{\gamma}_{T}^{2}$	Implied bound on $\|\mathrm{SV}\|$
GPT-2 Small	$14.58$	$\leq 14$
GPT-2 Medium	$11.33$	$\leq 11$

Appendix

Table 5: Conservative evaluation of the support-set bound eq. 11 in the setting of fig. 1 (

n=100

Experiment	Models	Dataset	Pairs	Training
Data-level ( fig. 1 )	GPT-2 S/M	HH	100	DPO, 1000 ep.
Data-level ( table 1 )	GPT-2 S/M	HH	1000	DPO, 1000 ep.
	Llama-3-8B	UF	1000	Full-FT DPO, 3 ep.
	Mistral-7B	UF	1000	Full-FT DPO, 3 ep.
Parameter-level ( fig. 2 )	GPT-2 M	HH	100	–
Parameter-level ( table 2 )	All five	UF	1000	–

Appendix

Table 6: Experimental settings. “Pairs” is the number of preference pairs on which each quantity is computed. HH: Anthropic HH-RLHF; UF: UltraFeedback Binarized; ep.: epochs.

Epoch	logps/chosen	logps/rejected	Gap (chosen $-$ rejected)
1	$-247.2$	$-258.9$	$11.7$
100	$-280.5$	$-334.5$	$54.0$
1000	$-308.3$	$-436.9$	$128.6$

Appendix

Table 7: Average log-probabilities of preferred (chosen) and dispreferred (rejected) responses during DPO training (GPT-2 Medium, HH-RLHF,

N=100

		Single		Merged		$\Delta$
Task Pair	Method	Task 1	Task 2	Task 1	Task 2	Task 1	Task 2
HH help $\leftrightarrow$ harm	SFT	$0.631$	$0.625$	$0.521$	$0.523$	$0.110$	$0.102$
HH help $\leftrightarrow$ harm	DPO	$0.668$	$0.645$	$0.641$	$0.619$	$\mathbf{0.027}$	$\mathbf{0.026}$
HH help $\leftrightarrow$ UF	SFT	$0.631$	$0.535$	$0.518$	$0.505$	$0.096$	$0.030$
HH help $\leftrightarrow$ UF	DPO	$0.668$	$0.572$	$0.632$	$0.558$	$\mathbf{0.036}$	$\mathbf{0.014}$

Appendix

Table 8: Single-task and merged performance of GPT-2 under task-vector merging.

\Delta

is the single-task minus merged performance (lower is better).

Asset	Version (Hugging Face)	License
GPT-2 Small	openai-community/gpt2	Modified MIT
GPT-2 Medium	openai-community/gpt2-medium	Modified MIT
Llama-3-8B-Instruct	meta-llama/Meta-Llama-3-8B-Instruct	Llama 3 Community License
Mistral-7B-Instruct-v0.2	mistralai/Mistral-7B-Instruct-v0.2	Apache 2.0
Gemma-2-9B-it	google/gemma-2-9b-it	Gemma Terms of Use
Anthropic HH-RLHF	Anthropic/hh-rlhf	MIT

Appendix

Table 9: Existing assets, versions, and licenses.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 165 papers

Aug 27, 2026

Disentangling Optimization Scale from Preference Scale in DPO

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient

β

commonly interpreted as controlling the KL constraint to a reference policy. We show that

β

entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in

β

: it vanishes in a dead zone at small

β

, reaches a peak at an intermediate value, and decreases again for larger

β

. Moreover, standard DPO loss values are not comparable across

β

: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of

β

, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for

β>0

, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous

β\to0

endpoint that reduces to a linear preference-margin objective.

Ivan Kruzhilov

Direct Preference Optimization Preference Learning Deep Learning Optimization RL for Language Models

Ask PDF

Mar 18, 2026

Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization

Direct Preference Optimization (DPO) has emerged as a popular algorithm for aligning pretrained large language models with human preferences, owing to its simplicity and training stability. However, DPO suffers from the recently identified squeezing effect (also known as likelihood displacement), where the probability of preferred responses decreases unintentionally during training. To understand and mitigate this phenomenon, we develop a theoretical framework that models the coordinate-wise dynamics in logit space. Our analysis reveals that negative-gradient updates cause residuals to expand rapidly along high-curvature directions, which underlies the squeezing effect, whereas Sharpness-Aware Minimization (SAM) can suppress this behavior through its curvature-regularization effect. Building on this insight, we investigate logits-SAM, a computationally efficient variant that perturbs only the output layer with negligible overhead. Extensive experiments on Pythia-2.8B, Mistral-7B, and Gemma-2B-IT across multiple datasets and benchmarks demonstrate that logits-SAM consistently improves the effectiveness of DPO and integrates seamlessly with other DPO variants. Code is available at https://github.com/RitianLuo/logits-sam-dpo.

Haocheng Luo, Zehang Deng, Thanh-Toan Do +3

Monash University, Australia · Swinburne University of Technology, Australia

LLM Alignment Direct Preference Optimization Sharpness-Aware Minimization

Ask PDF

Oct 6, 2026

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.

Yang Qu, Yusheng Han, Chengjia Feng +1

Northeastern University

Direct Preference Optimization LLM Alignment Preference Learning

Ask PDF

Abstract

Direct Preference Optimization

Explore similar work

Disentangling Optimization Scale from Preference Scale in DPO

Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization