Speculative Decoding (SD) has emerged as a premier technique for accelerating Large Language Model (LLM) inference by decoupling token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce Quasar (Quantized Self-speculative Acceleration for Rapid Inference), a novel, training-free framework designed to overcome this "memory wall" by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a 1.28× improvement in end-to-end throughput. Being orthogonal to existing drafting strategies, Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/Quasar.
Figures & tables
Figure 1: Comparison of the verification bottleneck in Self-Speculative Decoding versus Quasar. The left panel illustrates standard full-precision verification, where the process is memory-bound and saturates the memory bandwidth bus, leading to significant latency. The right panel depicts Quasar’s approach using low-bit quantized verification, which effectively reduces bandwidth pressure and enables high-throughput validation.
Figure 2: End-to-end speedup comparison across various benchmarks. We compare the acceleration achieved by Quasar against the Ngram baseline under two sampling settings: greedy decoding ( T=0 ) and stochastic sampling ( T=1 ). Quasar consistently outperforms the baseline across MT-bench, HumanEval, GSM8k, Alpaca, and CNN/DM, reaching up to a 1.6× speedup on reasoning-heavy tasks like GSM8k.
Task
MT-bench
HumanEval
GSM8k
Alpaca
CNN/DM
Overall
OpenPangu
Method
Speed
L
Speed
L
Speed
L
Speed
L
Speed
L
Speed
L
T=0
Vanilla
1.00×
1.00
1.00×
1.00
1.00×
1.00
1.00×
1.00
1.00×
1.00
1.00×
1.00
Ngram
1.04×
1.21
1.13×
1.36
1.23×
1.44
1.01×
1.16
1.03×
1.19
1.08×
1.27
Quasar
1.11×
1.25
1.22×
1.41
1.26×
1.46
1.06×
1.17
1.02×
1.15
1.13×
1.29
T=1
Vanilla
1.00×
1.00
1.00×
1.00
1.00×
1.00
1.00×
1.00
1.00×
1.00
1.00×
1.00
Table 1: The result of efficiency. Speed : Improvement in inference throughput. L : Average acceptance length (quality metric). T=0 / T=1 : Sampling temperatures. Results show that Quasar achieves the best speedup-quality trade-off.
Temperature
Ngram (BF16)
Quasar (W8A8)
Speed
L
Speed
L
T=0.0
1.18 ×
1.33
1.28 ×
1.40
T=0.2
1.17 ×
1.31
1.27 ×
1.39
T=0.4
1.16 ×
1.30
1.26 ×
1.38
T=0.6
1.15 ×
1.29
1.25 ×
1.36
T=0.8
1.14 ×
1.28
1.24 ×
1.35
Table 2: Robustness analysis across sampling temperatures ( T∈[0,1] ). Results are averaged over all tasks for Qwen3 . Speed : End-to-end inference speedup relative to Vanilla. L : Mean acceptance length. Quasar demonstrates superior stability, maintaining higher speedups as stochasticity increases.
K
Method
Metric
Speculative Tokens ( γ )
3
5
7
9
(1, 3)
Ngram
Speed
1.27 ×
1.28 ×
1.23 ×
1.23 ×
L
1.35
1.43
1.49
1.52
Quasar
Speed
1.34 ×
1.47 ×
1.45 ×
1.42 ×
L
1.44
1.46
1.49
1.59
(2, 4)
Ngram
Speed
1.14 ×
1.21 ×
1.28 ×
1.29 ×
Table 3: Detailed sensitivity analysis of Quasar vs. Ngram. K denotes the prompt lookup range, and γ is the number of speculative tokens. For each method, we report both the end-to-end Speedup and the Mean Acceptance Length ( L ) .
Benchmark (Acc ↑ )
OpenPangu-7B
Qwen3-8B
Score
Δ
Score
Δ
MMLU-pro
75.5
2.4%
63.1
2.5%
CEval
84.9
3.9%
83.4
1.3%
GPQA-Diamond
73.2
2.5%
62.0
3.1%
MATH-500
97.0
3.6%
87.4
2.1%
AIME24
79.3
3.4%
76.0
4.5%
Table 4: Scores comparison across serval benchmarks. For each model, we compare the original BF16 baseline with Quasar (W8A8). Results show Quasar achieves nearly identical performance to the baseline.
Method
Retention / Precision
L
Speedup
Vanilla (Full Model)
100% Layers / BF16
1.00
1.00 ×
Structural Pruning
Pruned-90%
90% Layers / BF16
1.62
0.80 ×
Pruned-75%
75% Layers / BF16
1.27
0.68 ×
Pruned-50%
50% Layers / BF16
1.03
0.62 ×
Ours
Table 5: Comparison between Structural Pruning and Quasar for Qwen3. α represents the token acceptance rate, and L denotes the mean acceptance length. Results highlight that training-free pruning fails to maintain the distributional alignment necessary for effective speculative decoding.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Running large language models (LLMs) locally continues to be limited by restrictions of compute and memory on consumer hardware. The popular acceleration technologies, such as quantization, speculative decoding, and adaptive inferencing, offer substantial speed boosts but usually necessitate retraining, per architecture tuning, or draft models. SpecQuant is a trainingfree framework, that combines speculative decoding with multiparent quantization to perform adaptive, efficient inference of LLMs. SpecQuant derives multiple quantized variants (INT4, FP8, FP16) from a shared base model, and dynamically routes queries based on predicted complexity; lightweight variants are used for simple or factual tasks, and full-precision models are used for complex reasoning tasks or long-context inputs. The shared-weight design of SpecQuant ensures sufficient token acceptance for speculative decoding without compatibility issues using separate draft parent models. We evaluate SpecQuant on Qwen2.5 based models on the MMLU, AlpacaEval, and GSM8K datasets, or benchmarks, demonstrating 35-43% speedups without degrading accuracy greater than 2%, substantial within the LLM community. SpecQuant enables practical on-device LLM deployment across diverse hardware without special infrastructure or expertise.
Harish KB, Jagadeeswaran M, Pradheep P +2
Computer Science and Engineering Vellore Institute of Technology Vellore, India
Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.
Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
We introduce PARSE (PArallel pRefix Speculative Engine), a speculative generation framework that accelerates large language model (LLM) inference by parallelizing prefix verification on a semantic level. Existing speculative decoding methods are fundamentally limited by token-level equivalence: the target model must verify each token, leading to short acceptance lengths and modest speedups. Moving to semantic or segment-level verification can substantially increase acceptance granularity, but prior approaches rely on sequential verification, introducing significant overhead and limiting practical gains. PARSE introduces parallel prefix verification, enabling semantic-level verification without sequential checks. Given a full draft from a draft model, the target model evaluates correctness across multiple prefixes in a single forward pass using a custom attention mask, directly identifying the maximal valid prefix. This eliminates sequential segment verification, and makes verification compute-efficient. PARSE is orthogonal to token-level speculative decoding and can be composed with it for additional gains. Across models and benchmarks, PARSE delivers 1.25× to 4.3× throughput gain over the target model, and 1.6× to 4.5× when composed with EAGLE-3, all with negligible accuracy degradation. This demonstrates parallel prefix verification as an effective, general approach to accelerating LLM inference.