In speculative decoding, parallel drafters such as DFlash make drafting cheap, so target verification dominates each round. A central choice is the verification length, the number of tokens that the target processes in each round. Verifying more draft tokens can accept more of them, but work after the first rejection is wasted. For hybrid target models that combine Gated DeltaNet (GDN) and mixture-of-experts (MoE) layers, verification time grows much faster with this length than for dense models. In a single-request profile of Qwen3.6-35B-A3B with DFlash, it grew by 71% from length 4 to 16, almost entirely in GDN and MoE layers, compared with 4%-6% for a dense Qwen3-4B target. Each GDN layer must update a recurrent state at every token and recover it at the end of the accepted prefix. The MoE cost depends on the experts that the tokens activate, which are known only once verification runs. We present HybridSpec, which selects the verification length on the GPU from draft-side acceptance estimates and measured target latency. Verification then runs this length plus one extra token up to the first MoE router, before any expert runs. There, a lightweight routing correction uses the observed expert assignments to decide whether to keep that token. The rest of verification runs only on the selected tokens. Block-parallel GDN kernels avoid storing a state after every token and reconstruct the accepted state of every GDN layer in one launch. In isolated benchmarks at verification lengths 4 to 16 with every draft token accepted, these kernels, including accepted-state reconstruction, are 1.54×-4.04× faster than the default kernels. With three GDN-MoE target models in single-request greedy decoding, HybridSpec is 1.75×-1.88× faster than autoregressive decoding and 1.30×-1.33× faster than DFlash.
Figures & tables
Figure 1: Profiled GPU time per verification call for Qwen3.6-35B-A3B with DFlash. (a) Times in ms, with target/drafter ratios in bold. (b) Bold percentages are increases from length 4 to 16.
Figure 2: One round of HybridSpec in single-request greedy decoding, with illustrative values: nominal verification length w0=6 , executed verification length w=7 , and A=4 accepted draft tokens. Circled numbers mark the steps described in Section 3.2 .
Figure 3: GDN verification and accepted-state reconstruction in the default SGLang verifier and in HybridSpec . The default verifier steps through the u tokens one at a time, whereas HybridSpec processes all u tokens in parallel and rebuilds the accepted states of all GDN layers in parallel, in one launch. Teal marks the accepted state and the factors of the m committed tokens.
Coding
Math
Reasoning
RAG
Summ.
Writing
Roleplay
Avg.
Method
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
Qwen3.5-35B-A3B
AR target
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
Vanilla DFlash
6.43
1.89×
5.40
1.54×
4.91
1.43×
6.07
1.72×
4.96
1.38×
3.92
1.13×
3.53
1.05×
5.03
1.45×
D-Cut
6.12
1.98×
4.97
1.64×
4.61
1.54×
5.76
1.80×
4.65
1.48×
3.70
1.26×
3.37
1.19×
4.74
1.55×
EVICT
6.18
1.99×
5.15
1.69×
4.68
1.58×
5.88
1.85×
4.78
1.54×
3.77
1.29×
3.36
1.20×
4.83
1.59×
Table 1: End-to-end comparison under greedy decoding. AL is the number of tokens emitted per verification round, and SR is the decode-throughput speedup over paired autoregressive (AR) decoding. Summ. denotes summarization, and Avg. is the mean over the seven categories.
Figure 4: Executed verification lengths of HybridSpec on the 280 test prompts. Vanilla DFlash verifies 16 tokens per round.
Table 6
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
w
Measured calls
Target total
GDN
MoE
Softmax attention
Other
4
101
9.527
2.498
5.331
0.686
1.019
5
85
10.055
2.659
5.691
0.696
1.017
6
80
10.811
2.835
6.248
0.710
1.026
7
74
11.876
2.984
7.171
0.705
1.026
8
72
12.648
3.096
7.826
0.698
1.037
10
71
14.075
3.413
8.934
0.705
1.033
Appendix
Table 4: Qwen3.6-35B-A3B target-side GPU time across verification lengths, in milliseconds per measured verification call. The total is the union of slightly overlapping component intervals, so the components do not sum exactly to it.
Latency (ms)
Speedup ( × ), m=w / m=⌈w/2⌉
w
SGLang
ReplaySSM
HybridSpec
vs. SGLang
vs. ReplaySSM
2
0.4726
0.6926
0.3552
1.33 / 1.35
1.95 / 1.66
3
0.5012
0.8058
0.3721
1.35 / 1.38
2.17 / 1.90
4
0.5920
0.9192
0.3847
1.54 / 1.62
2.39 / 1.90
8
0.9733
1.5050
0.4366
2.23 / 2.37
3.45 / 2.55
12
1.4263
2.4859
0.4491
3.18 / 3.38
5.54 / 4.59
Appendix
Table 5: Latency (ms) of GDN verification plus accepted-state reconstruction for the 30 GDN layers of Qwen3.6-35B-A3B on one A100 when all m=w input tokens are committed, that is, when every draft token is accepted. Speedups of HybridSpec are given for m=w and for m=⌈w/2⌉ , when about half of the input tokens are committed. ReplaySSM uses its default launch parameters, as in Table 1 .
Coding
Math
Reasoning
RAG
Summ.
Writing
Roleplay
Avg.
Method
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
AL
SR
Qwen3.5-35B-A3B
AR target
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
n/a
1.00×
Vanilla DFlash
5.29
1.56×
4.18
1.23×
3.88
1.15×
5.20
1.50×
4.11
1.19×
3.23
0.96×
2.63
0.81×
4.07
1.20×
D-Cut
4.70
1.58×
3.84
1.33×
3.78
1.32×
4.97
1.62×
3.88
1.31×
3.13
1.11×
2.54
0.94×
3.84
1.31×
EVICT
4.81
1.61×
3.97
1.37×
3.86
1.35×
5.03
1.63×
3.96
1.33×
3.17
1.13×
2.56
0.96×
3.91
1.34×
Appendix
Table 6: End-to-end comparison under temperature-one sampling, laid out as in Table 1 .
Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit.
Hongliang Cao, Xiuyu Li, Siqi Wang +3
Beihang University · Monash University · The Hong Kong University of Science and Technology
Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.
Bradley McDanel, Steven Li, Sruthikesh Surineni +1
Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share uˉ(n) of the per-slot bytes is below its ratio r(n) of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.