Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35% speedup over standard SD, with up to 50% fewer target model invocations while maintaining comparable accuracy.
Figures & tables
Figure 1: Comparison of (a) standard SD and (b-c) TriSpec. In (b), the proxy remains trusted at the first rejection, enabling local correction without invoking the target. In (c), the proxy becomes untrusted before the first rejection, so the remaining draft tokens are escalated to the target for authoritative verification. A margin-based criterion determines whether proxy verification is trusted.
Figure 2: (a) Latency decomposition of representative speculative decoding methods, with average per-round times shown on the left and acceptance lengths on the right. EAGLE reduces the drafting cost td , EAGLE-3 improves the acceptance length τ , and TriSpec reduces the average verification cost tv . Measurements use Qwen3-32B on GSM8K with batch size 1, a single A100 GPU, and draft length 6. (b) Token outcomes versus the proxy’s top-1–top-2 margin on ShareGPT. Exact matches agree with the target; mismatches passing the target-or-judge criterion (Appendix C.2 ) are benign; others are salient. Salient divergences cluster at low margins. Stacked areas are globally normalized.
Figure 3: (a) Draft-model and adapter training in TriSpec. The adapter maps proxy features into the drafter’s feature space; green, yellow, and gray modules denote target, proxy, and drafter components. (b) Token pruning and tree-attention mask updates after proxy-side verification in TriSpec.
GSM8K
MATH500
GK23-en
HumanEval
MBPP
Avg
Model
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Qwen3
Single Model (Target,32B)
98%
1.00 ×
85%
1.00 ×
75%
1.00 ×
87%
1.00 ×
77%
1.00 ×
84.4%
1.00 ×
Single Model (Proxy,1.7B)
86%
2.36 ×
81%
3.01 ×
70%
3.03 ×
81%
2.93 ×
51%
2.92 ×
73.8% -10.6%
2.85 ×
HASS
98%
2.98 ×
86%
2.27 ×
74%
2.30 ×
87%
2.03 ×
77%
2.18 ×
84.4% ± 0%
2.35 ×
HASS + SC[Chow]
80%
3.33 ×
55%
2.60 ×
51%
2.64 ×
74%
2.23 ×
56%
2.33 ×
63.2% -21.2%
2.63 ×
HASS + SC[OPT]
76%
3.25 ×
38%
2.66 ×
33%
2.65 ×
49%
2.30 ×
20%
2.34 ×
43.2% -41.2%
2.64 ×
Table 1: Accuracy and speedup on five reasoning benchmarks at temperature 0. “SC” denotes SpecCascade. In the Avg Acc. column, the small value denotes the change from target-only decoding.
GSM8K
MATH500
GK23-en
HumanEval
MBPP
Avg
Model
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Qwen3
Single Model (Target,32B)
98%
1.00 ×
85%
1.00 ×
76%
1.00 ×
87%
1.00 ×
76%
1.00 ×
84.4%
1.00 ×
Single Model (Proxy,1.7B)
88%
2.62 ×
80%
2.84 ×
73%
2.90 ×
76%
2.90 ×
61%
2.90 ×
75.6% -8.8%
2.83 ×
HASS
96%
2.61 ×
86%
2.05 ×
75%
2.05 ×
88%
1.96 ×
74%
1.94 ×
83.8% -0.6%
2.12 ×
HASS + TriSpec
95%
3.01 ×
84%
2.51 ×
77%
2.54 ×
89%
2.31 ×
72%
2.25 ×
83.4% -1.0%
2.52 ×
EAGLE3
98%
2.78 ×
84%
2.55 ×
77%
2.52 ×
86%
2.58 ×
74%
2.52 ×
83.8% -0.6%
2.59 ×
Table 2: Accuracy and speedup of different methods on five reasoning benchmarks, evaluated with a sampling temperature of 1.
SpecBench
HotpotQA
Polymath
AIME24
AIME25
GPQA-Diamond
Method
Score
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Qwen3
Single Model (Target, 32B)
8.82
1.00 ×
93.0%
1.00 ×
27.8%
1.00 ×
24/30
1.00 ×
22/30
1.00 ×
67.2%
1.00 ×
Single Model (Proxy, 1.7B)
7.33
2.72 ×
84.2%
2.96 ×
9.4%
3.35 ×
16/30
3.60 ×
12/30
3.54 ×
44.9%
3.33 ×
EAGLE3
8.77
2.80 ×
92.8%
2.77 ×
27.2%
2.34 ×
22/30
2.52 ×
21/30
2.31 ×
68.2%
2.97 ×
EAGLE3+TriSpec
8.62
3.17 ×
92.8%
3.58 ×
26.1%
3.11 ×
23/30
3.37 ×
21/30
3.30 ×
66.2%
4.39 ×
Table 3: Performance and speedup on general-domain and more challenging reasoning benchmarks.
MATH500
HumanEval
Model
Method
rt
tv (s)
τ
L (s)
Speed(Tok/s)
rt
tv (s)
τ
L (s)
Speed(Tok/s)
Qwen3
Single Model (Target,32B)
100.00%
/
/
258.76
14.24
100.00%
/
/
234.27
14.65
HASS
25.01%
0.094
4.21
115.04
32.45
27.14%
0.103
3.93
119.32
29.80
HASS + TriSpec
10.24%
0.070
4.39
83.03
45.26
11.61%
0.077
4.41
78.74
43.38
EAGLE3
21.38%
0.094
4.72
97.52
38.96
22.05%
0.103
4.60
86.62
39.76
EAGLE3 + TriSpec
9.95%
0.071
4.59
74.73
49.74
11.15%
0.080
4.76
71.83
48.00
Table 4: Per-sample averages of methods on MATH500 and HumanEval: Target-invocation ratio ( rt ), average per-round verification time ( tv ), average acceptance length ( τ ), Latency( L ), and Speed.
Figure 4: (a) Radar chart illustrating score and speed across all SpecBench subtasks. All experiments are conducted using models from the Qwen3 family. (b) Accuracy–speed trade-offs of different routing strategies. (c) Decoding speed under varying draft lengths for standard speculative decoding and TriSpec. All experiments are conducted on the MATH500 and using Qwen3-32B as the target.
MATH500
HumanEval
Training Strategy
TP
τ
rt
Speedup
τ
rt
Speedup
joint from scratch
1.77
4.59
9.95%
3.49 ×
4.76
11.1%
3.27 ×
adapter finetuning
0.42
4.52
9.99%
3.44 ×
4.65
11.4%
3.21 ×
Table 5: Ablations on the Qwen3 family. Left: comparing draft-model training strategies, with TP denoting trainable parameters in billions. Right: average performance with/without token pruning.
Figure 5: Accuracy–speed trade-off of TriSpec under different margin thresholds λ , where the dashed line denotes the target-only baseline accuracy.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Training Strategy
Data Size
Seq Len
TP(B)
Epochs
Training Time(h)
EAGLE3
/
518K
2048
1.35
6
78
EAGLE3 + TriSpec
joint-training from scratch
518K
2048
1.77
6
115
EAGLE3 + TriSpec
adapter-only finetune
518K
2048
0.42
3
44
Appendix
Table 6: Training cost of EAGLE3 and TriSpec on Qwen3 family
Component
2k
4k
8k
16k
32k
64k
128k
Target
66.01
66.50
67.47
69.42
73.32
81.12
96.72
Proxy
4.27
4.49
4.91
5.76
7.46
10.86
17.66
Drafter
4.16
4.16
4.18
4.21
4.28
4.41
4.67
Total
74.44
75.15
76.56
79.39
85.06
96.39
119.05
Proxy / Total
5.7%
6.0%
6.4%
7.3%
8.8%
11.3%
14.8%
Appendix
Table 7: GPU memory usage (GB) of Qwen3-series models under different context lengths.
GSM8K
MATH500
Gaokao2023en
Humaneval
MBPP
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Single Model (Target, 32B)
94.9%
1.00 ×
86.8%
1.00 ×
78.2%
1.00 ×
86.6%
1.00 ×
74.8%
1.00 ×
Single Model (Proxy, 1.7B)
86.0%
2.52 ×
78.2%
2.87 ×
69.9%
2.76 ×
77.4%
2.87 ×
52.8%
2.93 ×
EAGLE3
94.3%
3.29 ×
88.0%
2.77 ×
77.1%
2.77 ×
87.2%
2.72 ×
74.2%
2.71 ×
EAGLE3 + TriSpec
92.9%
4.08 ×
87.4%
3.48 ×
77.4%
3.39 ×
86.0%
3.23 ×
71.4%
3.33 ×
Appendix
Table 8: Accuracy and speedup results on the full benchmark suite for Qwen3 family
Figure 6: (a–b) Error bars indicate one standard deviation of accuracy and speed across repeated random subset evaluations for four generation methods. (c) Token-level exact-match ratios of Qwen3-family models at different parameter scales and the EAGLE-3 drafter, using Qwen3-32B predictions as the reference.
GSM8K
MATH500
Gaokao2023en
Humaneval
MBPP
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Single Model (Target, 8B)
64%
1.00 ×
34%
1.00 ×
18%
1.00 ×
66%
1.00 ×
57%
1.00 ×
Single Model (Proxy, 1B)
34%
2.54 ×
26%
2.53 ×
10%
2.50 ×
46%
2.58 ×
29%
2.53 ×
HASS
64%
2.93 ×
34%
3.27 ×
18%
3.23 ×
66%
3.52 ×
57%
3.46 ×
HASS + TriSpec
62%
3.17 ×
34%
3.73 ×
17%
3.66 ×
66%
3.81 ×
54%
3.76 ×
Appendix
Table 9: Accuracy and speedup results for original LLaMA3 family
Figure 7: An example of standalone inference for the three models in TriSpec.
Figure 8: Token-level comparison of the three models in TriSpec. Blue highlights mark benign divergences, and red highlights mark salient divergences. The reported matching ratios are measured over all ShareGPT samples.
Figure 9: Verification relaxation in the Qwen3 family on MATH500. (a) Confidence filtering under different confidence thresholds. (b) Top- k verification under different values of k .
Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.
Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronger target model to verify them. However, standard SD is mainly designed for acceleration, and its output quality is typically constrained by the target model. In this work, we propose Entropy-Aware Speculative Decoding (EASD), a lightweight and training-free extension of SD that improves reasoning quality through token-level entropy-guided rejection. EASD detects cases where both draft and target models exhibit high uncertainty while strongly overlapping in their top predictions. In such uncertain-agreement cases, EASD rejects the aligned token and resamples from the target distribution, preventing low-confidence errors from propagating. Experiments on challenging reasoning benchmarks show that EASD consistently improves accuracy over standard SD and reward-guided variants while maintaining comparable inference efficiency. Notably, EASD can surpass the standalone performance of the target model, suggesting that speculative decoding can serve not only as an acceleration method but also as an effective mechanism for improving reasoning quality. The code is available at https://github.com/ECNU-Text-Computing/EASD.
Tiancheng Su, Meicong Zhang, Guoxiu He
School of Economics and Management East China Normal University
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.
Shenggui Li, Chao Wang, Yikai Zhu +14
Nanyang Technological University · SpecForge Team · Meituan +2