NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies

Zhiyang Chen; Daliang Xu; Yinyuan Zhang; Chenghua Wang; Mengwei Xu; Yun Ma

cs.CLApr 8, 2026

NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies

Authors: Zhiyang Chen, Daliang Xu, Yinyuan Zhang, Chenghua Wang, Mengwei Xu, Yun Ma

Organizations: Institute for Artificial Intelligence, Peking University, Beijing, China · State Key Laboratory of Networking and Switching Technology; School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China · Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; School of Computer Science, Peking University, Beijing, China

Abstract

The massive vocabulary sizes of large language models, often exceeding 100k tokens, impose a computational bottleneck on the final linear projection layer during speculative decoding. Existing vocabulary pruning solutions rely on static or coarsely-grained sub-vocabularies that necessitate large active sizes ( $\sim$ 30k) to maintain draft quality. We propose NanoSpec, a novel training-free approach that breaks this trade-off by dynamically constructing a minimalist, context-aware active vocabulary for each generation step. Leveraging the inherent temporal locality of language generation, NanoSpec achieves high coverage while slashing the average vocabulary size by over $40\times$ (to $<$ 3k tokens) without requiring any auxiliary trained parameters. To realize the theoretical benefits of such high sparsity on modern hardware, we introduce a system-algorithm co-design that overcomes the inefficiencies of sparse memory access through asynchronous gathering and GPU-resident state management. As a complementary plug-and-play module, NanoSpec cuts draft time by an average of 51.6%, delivering a $1.17$ - $1.29\times$ end-to-end speedup over the state-of-the-art speculative decoding methods EAGLE-2 and EAGLE-3 across 7 tasks and outperforming complex training-based pruning baselines.

Figures & tables

Method	Pruning Strategy	Extra Training	Active Vocab.	EAGLE-2		EAGLE-3
Method	Pruning Strategy	Extra Training	Active Vocab.	Speed	Acc. Len.	Speed	Acc. Len.
Full Vocabulary	None	None	128k	336.9	3.80	362.2	4.25
FR-Spec	High-Frequency Tokens	None	32k	369.7	3.62	396.8	3.96
DynaSpec	Route to Fixed Clusters	1 MLP	27k	367.7	3.73	388.3	4.08
NanoSpec (Ours)	Dynamic by Context	None	$<\!3$ k	392.7	3.59	430.0	4.25

Table 1: Comparison of vocabulary pruning strategies ( Zhao et al., 2025 ; Zhang et al., 2025 ) for speculative decoding using Llama-3.1-8B-Instruct on SpecBench and HumanEval. Speed denotes generation speed (tokens/s); Acc. Len. denotes average acceptance length. NanoSpec achieves competitive acceptance across all baselines via dynamic vocabulary concentration (Section 4.2 ).

Panel of Figure 1 — Figure 1: Motivation analysis using Llama-3.1-8B. (a) The untapped potential in static vocabulary pruning. While theoretical acceleration exists for smaller vocabularies (oracle speed), actual speed drops rapidly due to reduced draft acceptance rates. Our target is to bridge this gap. (b) Vocabulary coverage analysis across diverse domains. Static method (dashed lines) requires large vocabularies for high coverage. Our approach ( NanoSpec , stars) achieves superior coverage with minimalist dynamic vocabulary (maximum of 3k tokens).

Figure 2: Overview of NanoSpec . Given the context “The old wooden ship had”, long-tail tokens “weathered, barnacle, hull” lie outside a fixed boundary, despite being highly probable in this specific context. Consequently, the draft model is forced to select generic, suboptimal high-frequency alternatives (“broken, surface”). NanoSpec recovers from this by dynamically constructing the vocabulary of draft model.

Model	Method	MT	Conv	RAG	Math	QA	Summ	Code	Avg. Speed	Avg. Len.
Llama-3.1 8B-Instruct	AR	175.7	178.5	165.0	178.2	174.2	174.3	162.3	172.6 (1.00 $\times$ )	1.00
	EAGLE-2	311.2	380.2	300.2	403.2	316.4	315.0	332.0	336.9 (1.93 $\times$ )	3.80
	FR-Spec	338.4	419.3	330.2	456.5	351.0	356.0	336.3	369.7 (2.12 $\times$ )	3.62
	DynaSpec	334.8	417.9	323.9	440.8	340.6	343.6	372.4	367.7 (2.13 $\times$ )	3.73
	NanoSpec	349.5	440.2	352.1	480.0	361.3	385.1	381.1	392.7 (2.25 $\times$ )	3.59
Llama-3.2 1B-Instruct	AR	675.6	677.2	624.7	677.9	672.2	661.7	671.3	665.8 (1.00 $\times$ )	1.00

Table 2: Comparison on end-to-end generation speed (tokens/s) and acceptance length across tasks and models. The baseline is standard auto-regressive decoding (AR). Avg. Speed is the mean across tasks (averaged over 3 runs), with relative speedup over AR in parentheses. Avg. Len. denotes the mean acceptance length per step. The best results for each model are bolded.

Figure 3: Performance comparison across tasks on Llama-3.1-8B-Instruct. Despite having shorter acceptance lengths, NanoSpec consistently outperforms all baselines in terms of both end-to-end speed and decoding speed.

Model	Method	MT	Conv	RAG	Math	QA	Summ	Code	Avg. Speed	Avg. Len.
Llama-3.1 8B-Instruct	EAGLE-3	308.6	412.4	332.5	440.9	322.0	339.6	379.7	362.2 (1.00 $\times$ )	4.25
	FR-Spec	345.5	456.8	365.0	501.7	359.5	388.4	360.6	396.8 (1.10 $\times$ )	3.96
	NanoSpec	356.7	489.2	393.0	530.2	381.3	442.3	417.5	430.0 (1.19 $\times$ )	4.25
Llama-3.2 1B-Instruct	EAGLE-3	654.4	805.1	675.5	881.9	671.8	616.7	871.6	739.6 (1.00 $\times$ )	3.45
	FR-Spec	817.8	987.5	817.3	1121.5	827.9	775.1	950.1	899.6 (1.22 $\times$ )	3.27
	NanoSpec	837.7	1029.0	846.7	1176.4	849.1	810.7	1064.0	944.8 (1.28 $\times$ )	3.39

Table 3: Compatibility with EAGLE-3: end-to-end speed (tokens/s) and acceptance length across tasks. NanoSpec delivers consistent speedups over vanilla EAGLE-3 without additional training.

Figure 4: Draft time comparison. NanoSpec effectively reduces draft overhead with a smaller vocabulary. The error bars indicate the minimum and maximum of average draft time.

Component	Full Vocab	FR-Spec	NanoSpec
Draft Backbone	1.405	1.401	1.403
Weight Gather	–	–	0.003
Draft LM Head	2.330	0.771	0.237
Total Draft Step	3.735	2.172	1.640

Table 4: Per-step latency breakdown (ms) of the draft model on Llama-3.1-8B-Instruct. The weight gather overhead of NanoSpec is fully hidden behind the concurrent backbone computation.

Vocab. Source	Async. Gather	Avg. Speed
Ctx. Only	$\checkmark$	313.3
Ext. Only	$\checkmark$	351.4
Ctx. + Ext.	$\times$ (Indexed GEMM)	364.1
Ctx. + Ext.	$\checkmark$	394.6

Table 5: Ablation study on average end-to-end speed across all SpecBench tasks using Llama-3.1-8B-Instruct.

Buffer	Purpose	Size
token_ids	GPU-resident token set index (i.e., $\mathcal{I}_{t}$ )	16 KB
tokens_tensor	Temporarily store new tokens (i.e., $\mathcal{C}_{draft}^{(t)},\mathcal{C}_{ver}^{(t)}$ )	12 KB
repack_buf	Dense buffer that repacks LM-head weight	24.00 MB
Total		24.02 MB

Table 6: VRAM overhead of NanoSpec on Llama-3.1-8B (FP16).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Average Acceptance Length by Task							Overall
Method	MT	Conv	RAG	Math	QA	Summ	Code	Overall
EAGLE-2 (Full Vocabulary)	3.03	3.86	3.70	4.31	3.06	3.43	4.19	3.65
FR-Spec	2.94	3.60	3.58	4.14	3.00	3.31	3.52	3.44
DynaSpec	2.86	3.72	3.32	4.18	2.97	3.24	3.96	3.46
NanoSpec	2.69	3.70	3.64	4.14	2.86	3.37	3.98	3.48

Appendix

Table 7: Average acceptance length per step on Qwen-2-7B-Instruct across different tasks. A higher value indicates better quality of speculative generation and effective vocabulary coverage. The results of DynaSpec are from its reported optimal performance.

Figure 5: Sensitivity analysis of the maximum of vocabulary size (

W_{max}

). The dual-axis plot shows the trade-off between average acceptance length (dashed blue, right axis) and end-to-end Speed (solid green, left axis) as

W_{max}

varies. The star marks our chosen default setting

W_{max}=3072

Vocabulary Source	Coverage (%)	Avg. Accept Length
Ctx. Only	64.23	2.89
Ext. Only	82.37	3.55
Ctx. + Ext. (Full NanoSpec )	91.65	3.59

Appendix

Table 8: Contribution of vocabulary sources to coverage and acceptance length on Llama-3.1-8B-Instruct (averaged across SpecBench tasks).

Figure 6: Per-category vocabulary evolution during decoding on Llama-3.1-8B-Instruct. Top row: active vocabulary size; bottom row: cumulative coverage (%). Solid lines denote the mean across samples; shaded regions indicate the min–max range. Each column corresponds to one task, with its typical decode length shown in the subtitle.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
Mar 5, 2026Ofir Ben ShohamLLM Inference Acceleration Speculative Decoding
Intuit
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
Apr 17, 2026Shuyu Zhang, Lingfeng Pan, Qicheng Wang +6Speculative Decoding LLM Inference Acceleration
School of Computer Science and Technology Xidian University Xi’an, China

Mar 5, 2026J/K move · Enter open · S save

Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding

Ofir Ben Shoham

Intuit

LLM Inference Acceleration Speculative Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates inference for Large Language Models by using a lightweight draft model to propose candidate tokens that are verified in parallel by a larger target model. Prior work shows that the draft model often dominates speculative decoding latency, since it generates tokens sequentially and incurs high cost from its language modeling head as vocabulary size grows. This exposes a fundamental trade-off in draft model design: larger vocabularies improve token coverage and agreement with the target model, but incur higher draft latency, while smaller vocabularies reduce latency at the risk of missing tokens required for accurate draft generation. We address this trade-off through vocabulary trimming for draft models, motivated by the observation that domain-specific workloads use only a small fraction of the full vocabulary. We cast draft vocabulary selection as a constrained optimization problem that balances token coverage and draft latency. Coverage is computed over assistant responses in the training data, while latency is estimated using architecture-aware FLOPs that capture the cost of the language modeling head as a function of vocabulary size. We optimize a utility function with a Tree-structured Parzen Estimator to efficiently explore the coverage-latency Pareto frontier under a minimum coverage constraint. Experiments show improved speculative decoding throughput while reducing draft vocabularies by up to 97% with high coverage. On domain-specific tasks, we achieve up to 16% latency reduction and 20% throughput improvement, and up to 6.7% throughput gains on diverse out-of-distribution tasks.

Abstract

Speculative Decoding

Explore similar work

Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation