Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35% speedup over standard SD, with up to 50% fewer target model invocations while maintaining comparable accuracy.
Figures & tables
Figure 1: Comparison of (a) standard SD and (b-c) TriSpec. In (b), the proxy remains trusted at the first rejection, enabling local correction without invoking the target. In (c), the proxy becomes untrusted before the first rejection, so the remaining draft tokens are escalated to the target for authoritative verification. A margin-based criterion determines whether proxy verification is trusted.
Figure 2: (a) Latency decomposition of representative speculative decoding methods, with average per-round times shown on the left and acceptance lengths on the right. EAGLE reduces the drafting cost td , EAGLE-3 improves the acceptance length τ , and TriSpec reduces the average verification cost tv . Measurements use Qwen3-32B on GSM8K with batch size 1, a single A100 GPU, and draft length 6. (b) Token outcomes versus the proxy’s top-1–top-2 margin on ShareGPT. Exact matches agree with the target; mismatches passing the target-or-judge criterion (Appendix C.2 ) are benign; others are salient. Salient divergences cluster at low margins. Stacked areas are globally normalized.
Figure 3: (a) Draft-model and adapter training in TriSpec. The adapter maps proxy features into the drafter’s feature space; green, yellow, and gray modules denote target, proxy, and drafter components. (b) Token pruning and tree-attention mask updates after proxy-side verification in TriSpec.
GSM8K
MATH500
GK23-en
HumanEval
MBPP
Avg
Model
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Qwen3
Single Model (Target,32B)
98%
1.00 ×
85%
1.00 ×
75%
1.00 ×
87%
1.00 ×
77%
1.00 ×
84.4%
1.00 ×
Single Model (Proxy,1.7B)
86%
2.36 ×
81%
3.01 ×
70%
3.03 ×
81%
2.93 ×
51%
2.92 ×
73.8% -10.6%
2.85 ×
HASS
98%
2.98 ×
86%
2.27 ×
74%
2.30 ×
87%
2.03 ×
77%
2.18 ×
84.4% ± 0%
2.35 ×
HASS + SC[Chow]
80%
3.33 ×
55%
2.60 ×
51%
2.64 ×
74%
2.23 ×
56%
2.33 ×
63.2% -21.2%
2.63 ×
HASS + SC[OPT]
76%
3.25 ×
38%
2.66 ×
33%
2.65 ×
49%
2.30 ×
20%
2.34 ×
43.2% -41.2%
2.64 ×
Table 1: Accuracy and speedup on five reasoning benchmarks at temperature 0. “SC” denotes SpecCascade. In the Avg Acc. column, the small value denotes the change from target-only decoding.
GSM8K
MATH500
GK23-en
HumanEval
MBPP
Avg
Model
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Qwen3
Single Model (Target,32B)
98%
1.00 ×
85%
1.00 ×
76%
1.00 ×
87%
1.00 ×
76%
1.00 ×
84.4%
1.00 ×
Single Model (Proxy,1.7B)
88%
2.62 ×
80%
2.84 ×
73%
2.90 ×
76%
2.90 ×
61%
2.90 ×
75.6% -8.8%
2.83 ×
HASS
96%
2.61 ×
86%
2.05 ×
75%
2.05 ×
88%
1.96 ×
74%
1.94 ×
83.8% -0.6%
2.12 ×
HASS + TriSpec
95%
3.01 ×
84%
2.51 ×
77%
2.54 ×
89%
2.31 ×
72%
2.25 ×
83.4% -1.0%
2.52 ×
EAGLE3
98%
2.78 ×
84%
2.55 ×
77%
2.52 ×
86%
2.58 ×
74%
2.52 ×
83.8% -0.6%
2.59 ×
Table 2: Accuracy and speedup of different methods on five reasoning benchmarks, evaluated with a sampling temperature of 1.
SpecBench
HotpotQA
Polymath
AIME24
AIME25
GPQA-Diamond
Method
Score
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Qwen3
Single Model (Target, 32B)
8.82
1.00 ×
93.0%
1.00 ×
27.8%
1.00 ×
24/30
1.00 ×
22/30
1.00 ×
67.2%
1.00 ×
Single Model (Proxy, 1.7B)
7.33
2.72 ×
84.2%
2.96 ×
9.4%
3.35 ×
16/30
3.60 ×
12/30
3.54 ×
44.9%
3.33 ×
EAGLE3
8.77
2.80 ×
92.8%
2.77 ×
27.2%
2.34 ×
22/30
2.52 ×
21/30
2.31 ×
68.2%
2.97 ×
EAGLE3+TriSpec
8.62
3.17 ×
92.8%
3.58 ×
26.1%
3.11 ×
23/30
3.37 ×
21/30
3.30 ×
66.2%
4.39 ×
Table 3: Performance and speedup on general-domain and more challenging reasoning benchmarks.
MATH500
HumanEval
Model
Method
rt
tv (s)
τ
L (s)
Speed(Tok/s)
rt
tv (s)
τ
L (s)
Speed(Tok/s)
Qwen3
Single Model (Target,32B)
100.00%
/
/
258.76
14.24
100.00%
/
/
234.27
14.65
HASS
25.01%
0.094
4.21
115.04
32.45
27.14%
0.103
3.93
119.32
29.80
HASS + TriSpec
10.24%
0.070
4.39
83.03
45.26
11.61%
0.077
4.41
78.74
43.38
EAGLE3
21.38%
0.094
4.72
97.52
38.96
22.05%
0.103
4.60
86.62
39.76
EAGLE3 + TriSpec
9.95%
0.071
4.59
74.73
49.74
11.15%
0.080
4.76
71.83
48.00
Table 4: Per-sample averages of methods on MATH500 and HumanEval: Target-invocation ratio ( rt ), average per-round verification time ( tv ), average acceptance length ( τ ), Latency( L ), and Speed.
Figure 4: (a) Radar chart illustrating score and speed across all SpecBench subtasks. All experiments are conducted using models from the Qwen3 family. (b) Accuracy–speed trade-offs of different routing strategies. (c) Decoding speed under varying draft lengths for standard speculative decoding and TriSpec. All experiments are conducted on the MATH500 and using Qwen3-32B as the target.
MATH500
HumanEval
Training Strategy
TP
τ
rt
Speedup
τ
rt
Speedup
joint from scratch
1.77
4.59
9.95%
3.49 ×
4.76
11.1%
3.27 ×
adapter finetuning
0.42
4.52
9.99%
3.44 ×
4.65
11.4%
3.21 ×
Table 5: Ablations on the Qwen3 family. Left: comparing draft-model training strategies, with TP denoting trainable parameters in billions. Right: average performance with/without token pruning.
Figure 5: Accuracy–speed trade-off of TriSpec under different margin thresholds λ , where the dashed line denotes the target-only baseline accuracy.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Training Strategy
Data Size
Seq Len
TP(B)
Epochs
Training Time(h)
EAGLE3
/
518K
2048
1.35
6
78
EAGLE3 + TriSpec
joint-training from scratch
518K
2048
1.77
6
115
EAGLE3 + TriSpec
adapter-only finetune
518K
2048
0.42
3
44
Appendix
Table 6: Training cost of EAGLE3 and TriSpec on Qwen3 family
Component
2k
4k
8k
16k
32k
64k
128k
Target
66.01
66.50
67.47
69.42
73.32
81.12
96.72
Proxy
4.27
4.49
4.91
5.76
7.46
10.86
17.66
Drafter
4.16
4.16
4.18
4.21
4.28
4.41
4.67
Total
74.44
75.15
76.56
79.39
85.06
96.39
119.05
Proxy / Total
5.7%
6.0%
6.4%
7.3%
8.8%
11.3%
14.8%
Appendix
Table 7: GPU memory usage (GB) of Qwen3-series models under different context lengths.
GSM8K
MATH500
Gaokao2023en
Humaneval
MBPP
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Single Model (Target, 32B)
94.9%
1.00 ×
86.8%
1.00 ×
78.2%
1.00 ×
86.6%
1.00 ×
74.8%
1.00 ×
Single Model (Proxy, 1.7B)
86.0%
2.52 ×
78.2%
2.87 ×
69.9%
2.76 ×
77.4%
2.87 ×
52.8%
2.93 ×
EAGLE3
94.3%
3.29 ×
88.0%
2.77 ×
77.1%
2.77 ×
87.2%
2.72 ×
74.2%
2.71 ×
EAGLE3 + TriSpec
92.9%
4.08 ×
87.4%
3.48 ×
77.4%
3.39 ×
86.0%
3.23 ×
71.4%
3.33 ×
Appendix
Table 8: Accuracy and speedup results on the full benchmark suite for Qwen3 family
Figure 6: (a–b) Error bars indicate one standard deviation of accuracy and speed across repeated random subset evaluations for four generation methods. (c) Token-level exact-match ratios of Qwen3-family models at different parameter scales and the EAGLE-3 drafter, using Qwen3-32B predictions as the reference.
GSM8K
MATH500
Gaokao2023en
Humaneval
MBPP
Method
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Acc.
Speedup
Single Model (Target, 8B)
64%
1.00 ×
34%
1.00 ×
18%
1.00 ×
66%
1.00 ×
57%
1.00 ×
Single Model (Proxy, 1B)
34%
2.54 ×
26%
2.53 ×
10%
2.50 ×
46%
2.58 ×
29%
2.53 ×
HASS
64%
2.93 ×
34%
3.27 ×
18%
3.23 ×
66%
3.52 ×
57%
3.46 ×
HASS + TriSpec
62%
3.17 ×
34%
3.73 ×
17%
3.66 ×
66%
3.81 ×
54%
3.76 ×
Appendix
Table 9: Accuracy and speedup results for original LLaMA3 family
Figure 7: An example of standalone inference for the three models in TriSpec.
Figure 8: Token-level comparison of the three models in TriSpec. Blue highlights mark benign divergences, and red highlights mark salient divergences. The reported matching ratios are measured over all ShareGPT samples.
Figure 9: Verification relaxation in the Qwen3 family on MATH500. (a) Confidence filtering under different confidence thresholds. (b) Top- k verification under different values of k .