On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

Kensuke Sasaki; Issei Sato

cs.LGOct 27, 2026

On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

Authors: Kensuke Sasaki, Issei Sato

Organizations: Department of Computer Science, The University of Tokyo

Abstract

Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement---a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7--9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to $35{\times}$ compared to supervised fine-tuning.

Figures & tables

Figure 1: Distribution of

A_{i}

across data pairs for GPT-2 Small (top) and Medium (bottom) at 1 epoch (left), 100 epochs (middle), and 1000 epochs (right) on HH-RLHF. Dark bars indicate the top 10% of pairs by weight. As training progresses, the distribution polarizes onto a small number of support pairs; this effect is more pronounced in the larger model.

Model	Dataset	Top-10% mass (%)
		Epoch 1	Epoch 100	Epoch 1000
GPT-2 Small	HH-RLHF	$10.88$	$28.78$	$39.57$
GPT-2 Medium	HH-RLHF	$11.02$	$32.32$	$43.70$
		Epoch 1	Epoch 2	Epoch 3
Llama-3-8B-Instruct	UltraFeedback	$11.75$	$41.82$	$52.48$
Mistral-7B-Instruct-v0.2	UltraFeedback	$11.23$	$39.92$	$43.77$

Table 1: Top-10% mass of

A_{i}

N=1000

pairs, i.e., the fraction of

\sum_{i}A_{i}

held by the top 10% of pairs (10% under a uniform distribution). Top: GPT-2 trained with DPO on HH-RLHF. Bottom: 7–8B models trained with full fine-tuning DPO on UltraFeedback Binarized.

Figure 2: Element-wise distribution of gradient feature vectors on a logarithmic scale for GPT-2 Medium on

N=100

pairs from HH-RLHF. (a) DPO: the vast majority of entries are near zero, confirming assumption 2 . (b) SFT: the distribution is substantially denser, since no gradient cancellation occurs. Near-zero fractions for all five models, computed in a separate setting, are reported in table 2 .

Model	Params	DPO $\Phi_{i}$ (%)	SFT $\Phi_{i}^{\mathrm{SFT}}$ (%)
GPT-2 Small	124M	$91.29$	$3.86$
GPT-2 Medium	355M	$94.82$	$4.34$
Llama-3-8B-Instruct	8B	$92.51$	$16.44$
Mistral-7B-Instruct-v0.2	7B	$90.93$	$19.72$
Gemma-2-9B-it	9B	$92.12$	$13.57$

Table 2: Fraction of near-zero entries in a representative gradient feature vector under DPO (

\Phi_{i}

) and SFT (

\Phi_{i}^{\mathrm{SFT}}

), computed from 1,000 pairs sampled from UltraFeedback Binarized. Higher values indicate sparser features.

Task Pair	Method	Model	Jaccard (%)	$\boldsymbol{\langle u_{1}^{\star},V_{2}\rangle}$
HH help $\leftrightarrow$ harm	SFT	Small	$54.9$	$9.12$
	DPO	Small	$18.9$	$0.258$
	SFT	Medium	$49.3$	$3.22$
	DPO	Medium	$11.4$	$0.191$
HH help $\leftrightarrow$ UF	SFT	Small	$68.4$	$21.2$
	DPO	Small	$25.2$	$1.91$

Table 3: Cross-task interference for DPO and SFT across two model scales (GPT-2 Small/Medium) and two task pairs: HH help

\leftrightarrow

harm (HH-RLHF helpful vs. harmless subsets) and HH help

\leftrightarrow

UF (HH-RLHF helpful vs. UltraFeedback). Lower values indicate greater disentanglement. Across all configurations, DPO reduces the interference

\langle u_{1}^{\star},V_{2}\rangle

11\times

35\times

relative to SFT.

Task Pair	Method	$\Delta$ Task 1	$\Delta$ Task 2
HH help $\leftrightarrow$ harm	SFT	$0.110$	$0.102$
HH help $\leftrightarrow$ harm	DPO	$\mathbf{0.027}$	$\mathbf{0.026}$
HH help $\leftrightarrow$ UF	SFT	$0.096$	$0.030$
HH help $\leftrightarrow$ UF	DPO	$\mathbf{0.036}$	$\mathbf{0.014}$

Table 4: Performance degradation

\Delta

(single-task

-

merged; lower is better) under task-vector merging on GPT-2 Medium. Task 1 is HH helpful; Task 2 is HH harmless or UltraFeedback. Full results are reported in appendix D .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\lambda_{\max}(K)/\tilde{\gamma}_{T}^{2}$	Implied bound on $\|\mathrm{SV}\|$
GPT-2 Small	$14.58$	$\leq 14$
GPT-2 Medium	$11.33$	$\leq 11$

Appendix

Table 5: Conservative evaluation of the support-set bound eq. 11 in the setting of fig. 1 (

n=100

Experiment	Models	Dataset	Pairs	Training
Data-level ( fig. 1 )	GPT-2 S/M	HH	100	DPO, 1000 ep.
Data-level ( table 1 )	GPT-2 S/M	HH	1000	DPO, 1000 ep.
	Llama-3-8B	UF	1000	Full-FT DPO, 3 ep.
	Mistral-7B	UF	1000	Full-FT DPO, 3 ep.
Parameter-level ( fig. 2 )	GPT-2 M	HH	100	–
Parameter-level ( table 2 )	All five	UF	1000	–

Appendix

Table 6: Experimental settings. “Pairs” is the number of preference pairs on which each quantity is computed. HH: Anthropic HH-RLHF; UF: UltraFeedback Binarized; ep.: epochs.

Epoch	logps/chosen	logps/rejected	Gap (chosen $-$ rejected)
1	$-247.2$	$-258.9$	$11.7$
100	$-280.5$	$-334.5$	$54.0$
1000	$-308.3$	$-436.9$	$128.6$

Appendix

Table 7: Average log-probabilities of preferred (chosen) and dispreferred (rejected) responses during DPO training (GPT-2 Medium, HH-RLHF,

N=100

		Single		Merged		$\Delta$
Task Pair	Method	Task 1	Task 2	Task 1	Task 2	Task 1	Task 2
HH help $\leftrightarrow$ harm	SFT	$0.631$	$0.625$	$0.521$	$0.523$	$0.110$	$0.102$
HH help $\leftrightarrow$ harm	DPO	$0.668$	$0.645$	$0.641$	$0.619$	$\mathbf{0.027}$	$\mathbf{0.026}$
HH help $\leftrightarrow$ UF	SFT	$0.631$	$0.535$	$0.518$	$0.505$	$0.096$	$0.030$
HH help $\leftrightarrow$ UF	DPO	$0.668$	$0.572$	$0.632$	$0.558$	$\mathbf{0.036}$	$\mathbf{0.014}$

Appendix

Table 8: Single-task and merged performance of GPT-2 under task-vector merging.

\Delta

is the single-task minus merged performance (lower is better).

Asset	Version (Hugging Face)	License
GPT-2 Small	openai-community/gpt2	Modified MIT
GPT-2 Medium	openai-community/gpt2-medium	Modified MIT
Llama-3-8B-Instruct	meta-llama/Meta-Llama-3-8B-Instruct	Llama 3 Community License
Mistral-7B-Instruct-v0.2	mistralai/Mistral-7B-Instruct-v0.2	Apache 2.0
Gemma-2-9B-it	google/gemma-2-9b-it	Gemma Terms of Use
Anthropic HH-RLHF	Anthropic/hh-rlhf	MIT

Appendix

Table 9: Existing assets, versions, and licenses.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 165 papers

Disentangling Optimization Scale from Preference Scale in DPO
Aug 27, 2026Ivan KruzhilovDirect Preference Optimization Preference Learning
Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization
Mar 18, 2026Haocheng Luo, Zehang Deng, Thanh-Toan Do +3LLM Alignment Direct Preference Optimization
Monash University, Australia · Swinburne University of Technology, Australia
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
Oct 6, 2026Yang Qu, Yusheng Han, Chengjia Feng +1Direct Preference Optimization LLM Alignment
Northeastern University

Aug 27, 2026J/K move · Enter open · S save

Disentangling Optimization Scale from Preference Scale in DPO

Ivan Kruzhilov

Direct Preference Optimization Preference Learning Deep Learning Optimization RL for Language Models

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient

β

commonly interpreted as controlling the KL constraint to a reference policy. We show that

β

entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in

β

: it vanishes in a dead zone at small

β

, reaches a peak at an intermediate value, and decreases again for larger

β

. Moreover, standard DPO loss values are not comparable across

β

: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of

β

, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for

β>0

, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous

β\to0

endpoint that reduces to a linear preference-margin objective.

Abstract

Direct Preference Optimization

Explore similar work

Disentangling Optimization Scale from Preference Scale in DPO

Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization