DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

Aaryan Ajay Sharma; Sai Nishanth Padala; Seganrasan Subramanian

cs.LGAug 28, 2026

DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

Authors: Aaryan Ajay Sharma, Sai Nishanth Padala, Seganrasan Subramanian

Organizations: ServiceNow · University of Twente

Abstract

Model merging combines multiple task-specific fine-tuned LLMs into a single multi-task model without additional training. However, merged models are known to suffer from representation bias: systematic drift between the merged model's hidden states and those of each individual source model. Prior work (Yang et al., 2024a) study and mitigate this bias for encoder-based vision models using a lightweight correction module trained with L1 loss. However, such bias is not studied for decoder models due to their autoregressive nature. We analyze the problem of representation bias in decoder models, and show two challenges absent in encoders: (1) the causal attention mask causes bias to accumulate across token positions, requiring position-dependent correction; and (2) not all token positions are equally important, i.e., high-entropy (decision-critical) positions matter far more than low-entropy ones. To address these challenges, we propose Decoder-Aware Representation Tuning via Surgery (DARTS). DARTS employs a novel entropy-weighted L1 loss to upweight correction at high-entropy positions where errors most affect generation quality, and a per-position additive bias that captures position-dependent error without overparameterization. We perform extensive evaluation on three domains: code generation (HumanEval), mathematical reasoning (GSM8K), and instruction following (AlpacaEval) on Llama-2-7B models, and show DARTS achieves significant improvement over the standard surgery approach while adding negligible parameters ( $0.1\%$ of total parameters).

Figures & tables

Figure 1: Per-token L1 representation bias between a merged Llama-2-7B model and each individual fine-tune, measured at the final transformer layer. Each row is a calibration sample; each column is a token position. Across all three domains (code, math, instruct), we see an increasing trend of bias with token position due to error accumulation under the causal attention mask. This position-dependent structure motivates the decoder-aware corrections proposed in DARTS .

Figure 2: Non-uniform importance of token positions in decoder models. We show three representative domains (math, code, instruction). For each example, we illustrate token-level entropy (

H

) and visualize position importance (higher entropy = higher importance). Injecting the same-magnitude representation bias

\delta

at low-importance positions has negligible impact, while injecting at high-importance positions causes significant performance degradation. This motivates entropy-weighted losses that focus corrections where they matter most.

Merging Method	Surgery Method	HumanEval	GSM8K	AlpacaEval	Avg.
Weight Averaging	None (baseline)	27.4	31.2	37.3	32.0
	Yang et al. (2024a)	26.2	28.7	39.5	31.5
	DARTS (ours)	26.8	28.9	41.3	32.3
Task Arithmetic	None (baseline)	23.8	41.2	49.5	38.2
	Yang et al. (2024a)	23.2	42.5	47.0	37.6
	DARTS (ours)	25.0	43.4	58.6	42.3

Table 1: Main results on three benchmarks. Comparing the original surgery approach (Standard +

L_{1}

) against our decoder-specific adaptations. Best results per merged model in bold .

Model	HE	GSM	AE
Llama-2-7B-Base	12.8	4.5	1.5
Llama-2-7B-Instruct	15.2	24.5	58.2
WizardMath-7B-v1.0	12.8	39.7	7.0
Llama-2-7B-Code	28.1	5.2	0.2

Table 2: Individual model baselines. HE = HumanEval (pass@1), GSM = GSM8K (exact match), AE = AlpacaEval (win rate %).

Merged Model	Domain	$\rho_{{L}_{1}}$	Task Score		pp
		(%)	No Surg.	DARTS	Change
TA ( $\lambda=0.5$ )	Code	8.2	23.8	25.0	+1.2
	Math	11.3	41.2	43.4	+2.2
	Instruct	42.9	49.5	58.6	+9.1
TA ( $\lambda=0.7$ )	Code	14.2	21.9	26.8	+4.9
	Math	14.3	46.2	42.9	$-$ 3.3

Table 3: Correlation between

L_{1}

bias reduction and task performance. pp = percentage point change in the task metric.

Seed	DARTS (Avg.)	Standard (Avg.)
default	42.3	37.6
7	42.2	41.7
29	42.3	40.4
71	43.3	41.8
164	42.2	41.2
190	43.0	41.1

Table 4: Statistical significance of DARTS vs. Standard surgery (

L_{1}

loss). Average score = mean of HumanEval, GSM8K, and AlpacaEval across 6 seeds.

Loss Function	HE	GSM	AE	Avg.
$L_{1}$ (original)	23.2	42.5	47.0	37.6
KL on logits	20.1	43.4	54.5	39.3
Cross-entropy distill.	18.3	33.1	40.0	30.5
$\text{EWL}_{1}$ (ours)	25.6	41.7	57.0	41.4

Table 5: Loss function ablation with Standard surgery. EWL1 dominates, confirming that loss choice is the primary driver of improvement.

Module	HE	GSM	AE
Standard	25.6	41.7	57.0
+ Position corr. (ours)	25.0	43.4	58.6
$\Delta$	$-$ 0.6	+1.7	+1.6

Table 6: Surgery module ablation with EWL1 loss. Position correction provides complementary gains on top of the improved loss.

Figure 3: Learning rate sensitivity of DARTS . Performance is stable across two orders of magnitude, with the default

10^{-3}

achieving the best average.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Total Samples	Calibration Split	Evaluation Split
HumanEval (Code)	164	50	114
GSM8K (Math)	1,319	50	1,269
AlpacaEval (Instruct)	805	50	755

Appendix

Table 7: Dataset partitions detailing the calibration (training) and evaluation (test) splits for each benchmark. The splits are mutually exclusive to prevent any potential leakage.

Phase	Yang et al. (2024a)	DARTS
Load merged model	3.37 s	3.46 s
Load target model	3.43 s	3.37 s
Precompute (2 $\times$ forward)	1.81 s	1.87 s
Training loop (500 iter)	2.40 s	3.15 s
Total wall-clock	11.01 s	11.85 s
Peak GPU memory	29.91 GB	31.49 GB

Appendix

Table 8: Per-domain calibration cost on a single NVIDIA A100 (80 GB). Model: Llama-2-7B-Code (

\approx

7B params), 50 calibration prompts,

T_{\max}=512

, rank 16, 500 iterations. The surgery computation (precompute + training) is

\approx

4–5 s; model loading (

\approx

7 s) dominates.

Calibration $N$	HE	GSM	AE
10	24.4	42.5	54.0
20	24.4	44.8	58.4
30	26.2	43.8	56.8
50	25.0	43.4	58.6
100	26.2	42.8	55.4
Full Test	23.17	43.52	59.45

Appendix

Table 9: Effect of calibration data size.

Loss Function	Surgery Module	HumanEval	GSM8K	AlpacaEval	Avg.
$L_{1}$	Standard	23.2	42.5	47.0	37.6
$L_{1}$	+ Pos. corr.	21.3	43.1	48.8	37.7
Entropy-Weighted $L_{1}$	Standard	25.6	41.7	57.0	41.4
Entropy-Weighted $L_{1}$	+ Pos. corr.	25.0	43.4	58.6	42.3

Appendix

Table 10: Full loss function

\times

surgery module grid.

Variant	HE	GSM	AE	Extra Params
Standard	25.6	41.7	57.0	0
Add	21.9	44.2	57.9	2.1M
Parallel	12.2	45.0	55.5	2.2M
Pos. Corr. (Ours)	25.0	43.4	58.6	2.1M

Appendix

Table 11: Position correction module variant comparison.

Steps	HE	GSM	AE
50	25.6	42.3	57.9
100	25.0	42.9	56.5
200	25.0	43.4	57.9
400	24.4	43.0	57.0
500 (default)	25.0	43.4	58.6
1000	25.6	44.3	58.4

Appendix

Table 12: Effect of training iterations.

Rank $r$	HE	GSM	AE	Params/domain
16	25.0	43.4	58.6	131K + 2.1M
32	25.0	44.3	56.4	262K + 2.1M
64	25.0	42.8	56.0	524K + 2.1M
128	23.8	42.6	58.1	1.04M + 2.1M

Appendix

Table 13: Effect of surgery rank on DARTS .

Seed	HE	GSM	AE
default	25.0	43.4	58.6
7	24.4	45.0	57.3
29	25.0	43.6	58.3
71	25.0	45.0	59.9
164	25.0	44.4	57.1
190	25.0	45.0	59.0

Appendix

Table 14: Reproducibility across random seeds. Seed controls the calibration data shuffle order.

Task	Mean Diff	$t$ -stat	$t$ -test $p$ -val	$W$ -stat	Wilcoxon $p$ -val
HumanEval	+0.30	0.690	0.521	7.500	0.688
GSM8K	+1.56	2.849	0.036	0.000	0.031
AlpacaEval	+3.95	2.459	0.057	0.000	0.031

Appendix

Table 15: Paired

t

-test and Wilcoxon signed-rank test comparing DARTS vs. Standard surgery across 6 random seeds for each task. Positive Mean Diff indicates DARTS outperforms the baseline.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 110 papers

DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees
Aug 13, 2026Tianyi Li, Yaxin Luo, Xinyi Shang +1Speculative Decoding Diffusion Language Model Decoding
VILA Lab, MBZUAI
DARE: Diffusion Language Model Activation Reuse for Efficient Inference
May 1, 2026Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang +3Self-Attention Diffusion Language Model Inference
Chandra Family Department of Electrical and Computer Engineering The University of Texas at Austin · Cornell University
DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training
Apr 18, 2026Ziwen Pan, Zihan Liang, Jad Kabbara +1LLM Auditing Social Bias in Language Models
Emory University · MIT

Aug 13, 2026J/K move · Enter open · S save

DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees

Tianyi Li, Yaxin Luo, Xinyi Shang, Zhiqiang Shen

VILA Lab, MBZUAI

Speculative Decoding Diffusion Language Model Decoding Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding losslessly accelerates autoregressive language models by verifying multiple draft tokens in parallel. Diffusion-based drafters further reduce proposal latency by predicting an entire token block in parallel, but their position-wise distributions are marginal rather than conditioned on tokens selected along each draft path. Existing recurrent correction incorporates causal information along a single draft chain, whereas diffusion-based tree construction broadens candidate coverage without carrying this correction along individual branches. We introduce DARTree, a training-free speculative decoding method that extends a pretrained AR correction head from chains to trees. DARTree first constructs a fixed-width candidate tree by expanding and scoring all nodes at each depth in a single batch, and then only applies best-first pruning to select the verification tree, decoupling AR-head inference from sequential heap operations. Across seven math, code, and chat benchmarks, DARTree achieves the highest average acceptance length and speedup in all four model--temperature configurations, accepting up to 12.97 tokens per verification round, 98.6% more than DFlash and 27.9% more than Domino in the same setting, and reaching up to 9.73

\times

lossless speedup over locally measured autoregressive decoding.

Abstract

Language Model Merging

Explore similar work

DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees

DARE: Diffusion Language Model Activation Reuse for Efficient Inference

DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training