TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Models

Haibo Hu; Lianming Huang; Qiao Li; Nan Guan; Chun Jason Xue

cs.CLAug 15, 2026

TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Models

Authors: Haibo Hu, Lianming Huang, Qiao Li, Nan Guan, Chun Jason Xue

Organizations: City University of Hong Kong · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Speculative decoding accelerates large language model inference through collaboration between a lightweight draft model and a target verifier. Existing methods mainly improve the draft side, while the target model is typically kept dense and unchanged. We show that, under domain-specific inference, full-depth target verification is not always the optimal choice. Counter-intuitively, skipping selected target layers can reduce verification cost while simultaneously increasing draft acceptance and preserving, or even improving, downstream task performance. Based on this observation, we propose TSS, a target-side sparsification framework for speculative decoding. TSS employs an acceptance- and metric-aware breadth search to explore multi-layer skip configurations without imposing a fixed priority between the two objectives. The selected configurations are stored in a domain-to-configuration mapping and applied by a lightweight skip controller, allowing one complete target model to support multiple sparse verification paths without retraining or permanent parameter pruning. Experiments on Spec-Bench across multiple domains, model scales, and speculative decoding methods show consistent improvements in draft acceptance and downstream task performance. In Translation setting, TSS increases the average accept length from 2.70 to 4.53 (+67.8%), improves BLEU from 0.131 to 0.237 (+80.9%), and raises end-to-end throughput from 75.6 to 127.3 tokens/s, corresponding to a 1.68X speedup.

Figures & tables

Figure 1: Comparison between conventional speculative decoding and TSS.

Figure 2: Mean accepted tokens per verification step for EAGLE and SAMD on Vicuna-7B and Llama-2-13B across four domains.

Figure 3: Single-layer skip ablation on Llama-2-13B with SAMD across four domains.

Figure 4: Overview of TSS. (a) Offline domain-wise profiling searches target-layer skip configurations using small calibration sets. (b) At runtime, a domain-aware controller retrieves the matched configuration and executes the corresponding sparse target-verification path, while leaving the draft model unchanged. (c) A representative case shows that TSS accepts more draft tokens and improves the final output over native dense verification.

Vicuna-7B (EAGLE / EAGLE+TSS, $L=32$ )
Domain	Method	Accept Len.	Task Metric	Skip Layers	Sparsity	Throughput(tok/s)	vs. Native
Translation	EAGLE	2.70	0.131	–	–	75.6	1.00 $\times$
	EAGLE+TSS	4.53	0.237	$\{3,25,30\}$	9.4%	127.3	1.68 $\times$
Summarization	EAGLE	3.93	0.266	–	–	89.4	1.00 $\times$
	EAGLE+TSS	4.01	0.270	$\{15,22,23\}$	9.4%	95.5	1.07 $\times$
RAG	EAGLE	3.24	0.096	–	–	76.0	1.00 $\times$

Table 1: The results on Spec-Bench.

Method	Accept Len.	Metric	Skip Layers	Throughput
Translation
SLEB	3.477	0.129	$\{2,3,4,9\}$	101.5
Random	3.382	0.140	$\{2,3,4,5\}$	100.3
GM-Skip	2.995	0.109	$\{12,14,22,23\}$	91.1
Accept-only	3.769	0.126	$\{3,15\}$	107.9
Metric-only	3.029	0.139	$\{12,18,28\}$	91.1

Table 2: Comparison of target-layer search strategies on MMLU and Translation, SLEB ( 26 ) and GM-Skip ( 11 ) .

Figure 5: Representative translation and MMLU cases. TSS produces an output closer to the translation reference and corrects an incorrect MMLU answer, while simultaneously improving accept length and throughput.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Setting	Value
Decoding	Precision	float16
	Temperature $\tau$	$0$
	Max new tokens $T$	$96$
	EAGLE total_token	$60$
Data	Random seed	$42$
	Selection train size	$16$ (QA-13B: $32$ )

Appendix

Table 3: Default hyperparameters for TSS selection and decoding.

Domain	Score Tol.	Accept Len.	Metric	Skip Layers	Sparsity	Tok/s	Evals
Translation	$+5\%$	4.134	0.1207	$\{4,9\}$	12.5%	120.7	351
	$0\%$	4.308	0.1402	$\{3,11,14,15\}$	12.5%	122.8	349
	$-5\%$	4.528	0.2370	{3,25,30}	9.4%	127.3	–
	$-10\%$	4.308	0.1402	$\{3,11,14,15\}$	12.5%	122.8	348
	$-20\%$	4.289	0.1462	$\{3,15,23,24\}$	12.5%	125.5	347
QA	$+5\%$	3.726	0.0600	$\{19,27\}$	12.5%	100.1	354

Appendix

Table 4: Sensitivity of TSS to the acceptance and metric tolerances on Vicuna-7B across four domains. We jointly vary

\epsilon_{A}

and

\epsilon_{M}

and report the resulting accept length, downstream metric, skip configuration, sparsity, throughput, and search evaluations. The

-5\%

setting provides the best overall balance between task quality and speculative acceptance.

Model	Domain	Request ID	Skip Layers	Metric	Native	+TSS
7B	Translation	sb_translation_192	$\{3,25,30\}$	BLEU	0.142	0.101
7B	Summarization	sum_289	$\{15,22,23\}$	ROUGE-L	0.281	0.430
7B	Open-domain QA	nq_57	$\{6,8,19,21,27\}$	F1	0.043	0.143
7B	RAG	sb_rag_513	$\{3,6,14,25,30\}$	F1	0.000	0.310
7B	MMLU	mmlu_prehistory_53	$\{3,7,9,14,20\}$	Acc.	0	0
13B	Translation	sb_translation_240	$\{7,11,20,31,35,38\}$	BLEU	0.100	0.328

Appendix

Table 5: Index of representative held-out qualitative cases across domains and model scales. The examples include improved, unchanged, and unsuccessful outcomes, illustrating both the benefits and limitations of domain-specific target-layer skipping.

Translation ( sb_translation_192 ); skip $\{3,25,30\}$ ; BLEU: 0.142 $\rightarrow$ 0.101
Input	Je dunkler das Fleisch , desto höher der ph-Wert .
Native	The darker the meat, the higher the pH value
+TSS	The darker the meat, the higher the pH
Ref.	The darker the meat , the higher the pH value .
Summarization ( sum_289 ); skip $\{15,22,23\}$ ; ROUGE-L: 0.281 $\rightarrow$ 0.430
Input	Arsenal must make a swift decision on Petr Cech this summer or risk losing out on the goalkeeper. The Chelsea shot-stopper looks certain to leave Stamford Bridge at the end of the season after losing the No 1 spot to Thibaut Courtois. Cech will hold talks wit…

Appendix

Table 6: Qualitative held-out examples for Vicuna-7B + EAGLE. Each block is one request from the official test split (

n{=}64

), comparing Native speculative decoding with the TSS skip set. Case-level task scores appear in each block header.

Translation ( sb_translation_240 ); skip $\{7,11,20,31,35,38\}$ ; BLEU: 0.100 $\rightarrow$ 0.328
Input	Ein Schuldspruch erfolgte außerdem in einer Reihe terrorismusbezogener Anklagepunkte , wozu eine Welle von Sprengstoffanschlägen auf Kirchen in Indonesien an Heiligabend im Jahr 2000 zählten .
Native	In addition to the sentence, a series of terrorism-related charges were brought, including a wave of bombings on churches in Indonesia on Christmas in the year 2000.
+TSS	A judgment was also handed down in a series of terrorism-related charges, including a wave of bombings on churches in Indonesia on Christmas Eve in the year 2000.
Ref.	He was also found guilty of a number of other terrorism-related charges , including a wave of bombings of churches across Indonesia on Christmas Eve in 2000 .
Summarization ( sum_246 ); skip $\{12,20,27,33,34,38\}$ ; ROUGE-L: 0.143 $\rightarrow$ 0.280
Input	Campaign: Amanda Burleigh has won her battle. A midwife has won a decade-long battle to give mothers more time attached to their newborns before the umbilical cord is cut. Amanda Burleigh was convinced that clamping the cord within seconds of delivery – shutt…

Appendix

Table 7: Qualitative held-out examples for Llama-2-13B + SAMD (Token Recycle). Each block is one request from the official test split (

n{=}64

), comparing Native speculative decoding with the TSS skip set. Case-level task scores appear in each block header.

Layer Skipping in Language Model Decoding

See all 11 papers

Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas
T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding
Oct 25, 2026Shahriar Kabir Nahin, Xiao Ye, Muhao Chen +2Speculative Decoding LLM Inference Acceleration
Bellini College of AI, Cybersecurity, and Computing, University of South Florida · School of Computing and Augmented Intelligence, Arizona State University · Department of Computer Science, University of California, Davis
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

May 24, 2026J/K move · Enter open · S save

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

Jinze Li, Yixing Xu, Guanchen Li, Jinfeng Xu+6

Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce Collaborative Speculative Decoding (CoSpec), a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.

Abstract

Layer Skipping in Language Model Decoding

Explore similar work

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification