Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification

Guang Huang; Zeyi Wen

cs.DCMar 2, 2026

Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification

Authors: Guang Huang, Zeyi Wen

Organizations: HKUST(GZ), Guangzhou, China · HKUST, Hong Kong, China

Abstract

Speculative Decoding (SD) has emerged as a premier technique for accelerating Large Language Model (LLM) inference by decoupling token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce Quasar (Quantized Self-speculative Acceleration for Rapid Inference), a novel, training-free framework designed to overcome this "memory wall" by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a $1.28\times$ improvement in end-to-end throughput. Being orthogonal to existing drafting strategies, Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/Quasar.

Figures & tables

Figure 1: Comparison of the verification bottleneck in Self-Speculative Decoding versus Quasar. The left panel illustrates standard full-precision verification, where the process is memory-bound and saturates the memory bandwidth bus, leading to significant latency. The right panel depicts Quasar’s approach using low-bit quantized verification, which effectively reduces bandwidth pressure and enables high-throughput validation.

Figure 2: End-to-end speedup comparison across various benchmarks. We compare the acceleration achieved by Quasar against the Ngram baseline under two sampling settings: greedy decoding ( $T=0$ ) and stochastic sampling ( $T=1$ ). Quasar consistently outperforms the baseline across MT-bench, HumanEval, GSM8k, Alpaca, and CNN/DM, reaching up to a $1.6\times$ speedup on reasoning-heavy tasks like GSM8k.

Task

MT-bench

HumanEval

GSM8k

Alpaca

CNN/DM

Overall

OpenPangu

Method

Speed

L

Speed

L

Speed

L

Speed

L

Speed

L

Speed

L

T=0

Vanilla

1.00×

1.00

1.00×

1.00

1.00×

1.00

1.00×

1.00

1.00×

1.00

1.00×

1.00

Ngram

1.04×

1.21

1.13×

1.36

1.23×

1.44

1.01×

1.16

1.03×

1.19

1.08×

1.27

Quasar

1.11×

1.25

1.22×

1.41

1.26×

1.46

1.06×

1.17

1.02×

1.15

1.13×

1.29

T=1

Vanilla

1.00×

1.00

1.00×

1.00

1.00×

1.00

1.00×

1.00

1.00×

1.00

1.00×

1.00

Table 1: The result of efficiency. Speed : Improvement in inference throughput.

L

: Average acceptance length (quality metric). T=0 / T=1 : Sampling temperatures. Results show that Quasar achieves the best speedup-quality trade-off.

Temperature	Ngram (BF16)		Quasar (W8A8)
Temperature	Speed	$L$	Speed	$L$
$T=0.0$	1.18 $\times$	1.33	1.28 $\times$	1.40
$T=0.2$	1.17 $\times$	1.31	1.27 $\times$	1.39
$T=0.4$	1.16 $\times$	1.30	1.26 $\times$	1.38
$T=0.6$	1.15 $\times$	1.29	1.25 $\times$	1.36
$T=0.8$	1.14 $\times$	1.28	1.24 $\times$	1.35

Table 2: Robustness analysis across sampling temperatures (

T\in[0,1]

). Results are averaged over all tasks for Qwen3 . Speed : End-to-end inference speedup relative to Vanilla. L : Mean acceptance length. Quasar demonstrates superior stability, maintaining higher speedups as stochasticity increases.

$K$	Method	Metric	Speculative Tokens ( $\gamma$ )
$K$	Method	Metric	3	5	7	9
(1, 3)	Ngram	Speed	1.27 $\times$	1.28 $\times$	1.23 $\times$	1.23 $\times$
	Ngram	$L$	1.35	1.43	1.49	1.52
	Quasar	Speed	1.34 $\times$	1.47 $\times$	1.45 $\times$	1.42 $\times$
	Quasar	$L$	1.44	1.46	1.49	1.59
(2, 4)	Ngram	Speed	1.14 $\times$	1.21 $\times$	1.28 $\times$	1.29 $\times$

Table 3: Detailed sensitivity analysis of Quasar vs. Ngram.

K

denotes the prompt lookup range, and

\gamma

is the number of speculative tokens. For each method, we report both the end-to-end Speedup and the Mean Acceptance Length (

L

) .

Benchmark (Acc $\uparrow$ )	OpenPangu-7B		Qwen3-8B
Benchmark (Acc $\uparrow$ )	Score	$\Delta$	Score	$\Delta$
MMLU-pro	75.5	2.4%	63.1	2.5%
CEval	84.9	3.9%	83.4	1.3%
GPQA-Diamond	73.2	2.5%	62.0	3.1%
MATH-500	97.0	3.6%	87.4	2.1%
AIME24	79.3	3.4%	76.0	4.5%

Table 4: Scores comparison across serval benchmarks. For each model, we compare the original BF16 baseline with Quasar (W8A8). Results show Quasar achieves nearly identical performance to the baseline.

Method	Retention / Precision	$L$	Speedup
Vanilla (Full Model)	100% Layers / BF16	1.00	1.00 $\times$
Structural Pruning
Pruned-90%	90% Layers / BF16	1.62	0.80 $\times$
Pruned-75%	75% Layers / BF16	1.27	0.68 $\times$
Pruned-50%	50% Layers / BF16	1.03	0.62 $\times$
Ours

Table 5: Comparison between Structural Pruning and Quasar for Qwen3.

\alpha

represents the token acceptance rate, and

L

denotes the mean acceptance length. Results highlight that training-free pruning fails to maintain the distributional alignment necessary for effective speculative decoding.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,080 papers

SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference
Sep 18, 2026Harish KB, Jagadeeswaran M, Pradheep P +2LLM Inference LLM Inference Acceleration
Computer Science and Engineering Vellore Institute of Technology Vellore, India
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
Parallel Prefix Verification for Speculative Generation
May 5, 2026Yuncheng Yao, Yuxuan Xia, Shengjie Wang +1Speculative Decoding LLM Inference Acceleration
Duke University · New York University

Sep 18, 2026J/K move · Enter open · S save

SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference

Harish KB, Jagadeeswaran M, Pradheep P, Yuvanesh S+1

Computer Science and Engineering Vellore Institute of Technology Vellore, India

LLM Inference LLM Inference Acceleration On-Device Language Model Inference LLM Quantization Speculative Decoding

Open paper AskPDF arXiv

Running large language models (LLMs) locally continues to be limited by restrictions of compute and memory on consumer hardware. The popular acceleration technologies, such as quantization, speculative decoding, and adaptive inferencing, offer substantial speed boosts but usually necessitate retraining, per architecture tuning, or draft models. SpecQuant is a trainingfree framework, that combines speculative decoding with multiparent quantization to perform adaptive, efficient inference of LLMs. SpecQuant derives multiple quantized variants (INT4, FP8, FP16) from a shared base model, and dynamically routes queries based on predicted complexity; lightweight variants are used for simple or factual tasks, and full-precision models are used for complex reasoning tasks or long-context inputs. The shared-weight design of SpecQuant ensures sufficient token acceptance for speculative decoding without compatibility issues using separate draft parent models. We evaluate SpecQuant on Qwen2.5 based models on the MMLU, AlpacaEval, and GSM8K datasets, or benchmarks, demonstrating 35-43% speedups without degrading accuracy greater than 2%, substantial within the LLM community. SpecQuant enables practical on-device LLM deployment across diverse hardware without special infrastructure or expertise.

Abstract

LLM Inference Acceleration

Explore similar work

SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Parallel Prefix Verification for Speculative Generation