SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Qitong Wang; Xinwei Niu; Mingluo Su; Shanwei Zhao; Shiai Zhu; Huan Wang

cs.LGOct 8, 2026

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Authors: Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang

Organizations: Westlake University · Ant Group

Abstract

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.

Figures & tables

Panel of Figure 1 — Figure 1: Motivating observations for decoding-aware sparse inference. (a) The relative activation discrepancy between the dense and pruned models grows rapidly during the first few decoding steps before settling into a plateau. The shaded region shows the interquartile range across transformer layers, while the curves correspond to the first (layer 0), middle (layer 16), and last (layer 31) layers. All layers reach 90% of their plateau by decoding step 12, after which the activation discrepancy remains largely unchanged. (b) Speedup of the $2{:}4$ sparse Tensor Core path over dense execution. Existing sparse kernels provide $1.31$ to $1.46\times$ speedups during the prefill stage, but achieve only $0.85$ to $0.87\times$ of dense performance during autoregressive decoding. Results are obtained on Llama-3.1-8B with $2{:}4$ sparsity on an NVIDIA A100 GPU.

Figure 2: Overview of SparseDecoding. Unlike standard pruning methods collecting calibration activations from fixed corpora, SparseDecoding calibrates on the model-generated tokens from task prompts while discarding the prefill activations. Meanwhile, existing

N{:}M

sparse Tensor Core kernels accelerate prefill but often underperform dense execution during autoregressive decoding. SparseDecoding instead introduces a decoding-oriented sparse execution pipeline that converts pruned weights into an efficient

N{:}M

representation and employs an optimized sparse decoding kernel, achieving up to

1.48\times

decoding speedup under the experimental setup described in Sec. 4 .

Model	Sparsity	Method	Overall	Domains						Requirements
Model	Sparsity	Method	Overall	D1	D2	D3	D4	D5	D6	R1	C	R2	C	R3	C
Llama-3.1-8B	0%	Dense	3.70	3.7	3.6	3.5	3.3	4.1	4.3	3.8	3.9	3.7	4.3	3.8	3.6
	$50\%$	C4	2.25	2.4	2.2	2.1	1.9	2.4	2.6	2.2	2.2	2.2	2.5	2.2	2.1
	$50\%$	SparseDecoding	2.54	2.7	2.5	2.5	2.0	2.7	3.0	2.5	2.5	2.5	2.9	2.5	2.4
	$2{:}4$	C4	1.34	1.5	1.3	1.3	1.1	1.4	1.6	1.3	1.3	1.3	1.4	1.3	1.2
	$2{:}4$	SparseDecoding	1.52	1.7	1.6	1.5	1.3	1.5	1.6	1.5	1.5	1.5	1.5	1.4	1.4

Table 1: Results on the WritingBench benchmark under

50\%

and

2{:}4

sparsity. SparseDecoding (ours) calibrates on autoregressive generations conditioned on LongWriter prompts; C4 uses fixed C4 sequences. Dense is the unpruned reference. Columns report the overall score, the six domain scores, and the three requirement scores for style (R1), format (R2), and length (R3); “C” indicates the corresponding category-specific score. The six domains are Academic & Engineering (D1), Finance & Business (D2), Politics & Law (D3), Literature & Arts (D4), Education (D5), and Advertising & Marketing (D6). All scores are on a scale of 1-10. Bold marks the better overall score between SparseDecoding and C4 within each sparsity setting, based on unrounded values.

Model	Sparsity	Method	Pass@1 (%) $\uparrow$
Llama-3.1-8B	0%	Dense	24.0
	50%	C4	2.0
	50%	SparseDecoding	10.0
	$2{:}4$	C4	0.0
	$2{:}4$	SparseDecoding	4.0
Llama-3.3-70B	0%	Dense	33.0

Table 2: Results on the ClassEval benchmark under

50\%

and

2{:}4

sparsity.

$N{:}M$	Llama-3.1-8B		Llama-3.3-70B		Qwen3-14B		Qwen3-32B
$N{:}M$	Token/s	Speedup	Token/s	Speedup	Token/s	Speedup	Token/s	Speedup
Dense	96.26	1.00 $\times$	21.67	1.00 $\times$	39.76	1.00 $\times$	19.15	1.00 $\times$
$2{:}4$	136.30	1.42 $\times$	30.17	1.39 $\times$	53.18	1.34 $\times$	27.69	1.45 $\times$
$4{:}8$	136.43	1.42 $\times$	31.64	1.46 $\times$	53.86	1.35 $\times$	27.83	1.45 $\times$
$8{:}16$	136.89	1.42 $\times$	29.90	1.38 $\times$	53.42	1.34 $\times$	27.77	1.45 $\times$
$16{:}32$	136.89	1.42 $\times$	32.13	1.48 $\times$	53.73	1.35 $\times$	27.81	1.45 $\times$

Table 3: End-to-end decoding throughput and speedup over the dense baseline . All of the

N{:}M

patterns retain

50\%

of the weights, so differences reflect kernel efficiency rather than compression. Every evaluated pattern decodes faster than the dense baseline, with speedups ranging from

1.34

1.48\times

. Measurements use a context length of 512 and report the median after 50 warmup and 50 measured iterations on NVIDIA A100 GPUs with GPT-Fast.

Model	Sparsity	Calibration	Overall
Qwen3-14B	$50\%$	Cross-model (32B)	5.23
	$50\%$	Self-generated (14B)	5.35
	$2{:}4$	Cross-model (32B)	4.13
	$2{:}4$	Self-generated (14B)	4.18

Table 4: Results on the WritingBench benchmark under self-generated and cross-model calibration. Qwen3-14B is pruned using activations from its own generations (self-generated) or from Qwen3-32B generations (cross-model) .

Model	Sparsity	Method	Overall
Qwen3-14B	$50\%$	C4	4.94
	$50\%$	SparseDecoding	5.23
	$2{:}4$	C4	2.41
	$2{:}4$	SparseDecoding	3.30
Qwen3-32B	$50\%$	C4	5.52
	$50\%$	SparseDecoding	5.71

Table 5: Results on the WritingBench benchmark under 50% and 2:4 sparsity with Wanda on the Qwen3 model family.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Retained eigenvectors	Modules with $\hat{\epsilon}_{\operatorname{AR}}<\hat{\epsilon}_{\operatorname{C4}}$	Percentage	Median ratio $\hat{\epsilon}_{\operatorname{C4}}/\hat{\epsilon}_{\operatorname{AR}}$
1%	274	97.9%	2.30
2%	271	96.8%	2.39
5%	266	95.0%	2.65
10%	264	94.3%	2.83
20%	261	93.2%	3.06
50%	248	88.6%	3.26

Appendix

Table 6: Empirical comparison of AR and C4 calibration discrepancy on Qwen3 14B. For each retained eigenvector fraction, we report the number and percentage of prunable modules (out of 280) where

\hat{\epsilon}_{\ell,\operatorname{AR}}^{(k)}<\hat{\epsilon}_{\ell,\operatorname{C4}}^{(k)}

, together with the median of the discrepancy ratio

\hat{\epsilon}_{\operatorname{C4}}/\hat{\epsilon}_{\operatorname{AR}}

Retained eigenvectors	Modules with $\hat{\epsilon}_{\operatorname{AR}}<\hat{\epsilon}_{\operatorname{C4}}$	Percentage	Median ratio $\hat{\epsilon}_{\operatorname{C4}}/\hat{\epsilon}_{\operatorname{AR}}$
1%	390	87.1%	1.81
2%	419	93.5%	1.92
5%	440	98.2%	2.08
10%	439	98.0%	2.16
20%	444	99.1%	2.17
50%	442	98.7%	2.19

Appendix

Table 7: Empirical comparison of AR and C4 calibration discrepancy on Qwen3 32B. For each retained eigenvector fraction, we report the number and percentage of prunable modules (out of 448) where

\hat{\epsilon}_{\ell,\operatorname{AR}}^{(k)}<\hat{\epsilon}_{\ell,\operatorname{C4}}^{(k)}

, together with the median of the discrepancy ratio

\hat{\epsilon}_{\operatorname{C4}}/\hat{\epsilon}_{\operatorname{AR}}

Model	Sparsity	Method	Overall	Domains						Requirements
Model	Sparsity	Method	Overall	D1	D2	D3	D4	D5	D6	R1	C	R2	C	R3	C
Qwen3-14B	$0\%$	Dense	5.97	6.0	5.8	6.1	5.7	6.3	6.1	6.1	6.4	6.0	6.8	6.0	6.3
	$50\%$	C4	4.62	4.6	4.4	4.6	4.0	5.4	5.2	4.8	5.0	4.6	5.2	4.5	4.6
	$50\%$	SparseDecoding	5.35	5.3	5.3	5.4	4.9	5.9	5.6	5.5	5.8	5.4	6.2	5.4	5.6
	$8{:}16$	C4	4.50	4.5	4.4	4.5	3.8	5.0	5.2	4.6	4.8	4.5	5.0	4.3	4.2
	$8{:}16$	SparseDecoding	5.31	5.2	5.3	5.3	4.7	5.9	5.7	5.4	5.6	5.4	6.1	5.2	5.4

Appendix

Table 8: Results on WritingBench for Qwen3-14B and Qwen3-32B under unstructured

50\%

and semi-structured

8{:}16

sparsity. SparseDecoding (ours) calibrates on autoregressive generations conditioned on LongWriter prompts; C4 uses fixed C4 sequences. Dense is the unpruned reference. “C” indicates the corresponding category-specific score. Bold marks the best overall score within each sparsity setting.

Model	Sparsity	Calibration	Overall	Domains						Requirements
Model	Sparsity	Calibration	Overall	D1	D2	D3	D4	D5	D6	R1	C	R2	C	R3	C
Qwen3-14B	$0\%$	Dense	5.97	6.0	5.8	6.1	5.7	6.3	6.1	6.1	6.4	6.0	6.8	6.0	6.3
	$50\%$	Pile	5.00	5.2	5.1	5.1	4.0	5.7	5.3	5.0	5.2	5.0	5.6	4.8	5.0
		RedPajama	5.04	5.0	5.0	5.1	4.4	5.6	5.6	5.2	5.4	5.1	5.7	4.9	5.0
		SparseDecoding	5.35	5.3	5.3	5.4	4.9	5.9	5.6	5.5	5.8	5.4	6.2	5.4	5.6
	$2{:}4$	Pile	2.68	3.0	2.7	2.6	1.9	3.0	3.1	2.7	2.7	2.7	2.9	2.4	2.2

Appendix

Table 9: Results on WritingBench benchmark under

50\%

and

2{:}4

sparsity with different calibration data for the Qwen3 model family. Pile and RedPajama use fixed pretraining-corpus sequences; SparseDecoding calibrates on autoregressive generations conditioned on LongWriter prompts.

Explore similar work

Jul 24, 2026cs.DC

Unified Static-Dynamic Pruning for Efficient LLM Inference

The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low compute intensity and bandwidth-bound kernels dominate inference cost. Weight pruning offers a promising remedy, but existing methods remain confined to either static pruning (SP), which permanently removes redundant weights but lacks adaptivity, or dynamic pruning (DP), which adapts to input sparsity but introduces runtime irregularity. This paper presents SPDP, a unified sparse-inference framework that integrates unstructured SP with input-adaptive DP for efficient LLM inference on GPUs. SPDP co-designs a new Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format and two complementary GPU kernels: (1) a CUDA-core spMspV kernel featuring Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for fine-grained, runtime activation skipping, and (2) a Tensor-Core SpMM kernel optimized for prefill computation. This joint format-kernel design harmonizes static and dynamic sparsity, maintaining bandwidth-efficient memory access and high compute intensity under both phases of LLM inference. Comprehensive evaluations on inference-optimized GPUs demonstrate that SPDP achieves 1.24x-1.37x average speedup (up to 2.51x) over state-of-the-art sparse frameworks such as SpInfer, while matching perplexity with up to 25% higher sparsity. SPDP advances the inference efficiency-quality Pareto frontier, showing that unified static-dynamic pruning can deliver substantial throughput and performance-per-watt improvements in large-scale LLM serving.

Jinhyeok Kim, Yejoon Lee, Jaeyoung Do

Seoul National University Seoul, South Korea

Efficient Neural Network Inference LLM Pruning LLM Inference Acceleration Neural Network Pruning

Ask PDF

Jun 9, 2026cs.LG

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Snowflake AI Research* · Seoul National University†

LLM Pruning High-Performance Computing Structured Sparsity LLM Inference Acceleration Neural Network Pruning

Ask PDF

Sep 16, 2026cs.AI

Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning

We introduce Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning. Existing selection strategies rely predominantly on scalar probabilities, ignoring geometric semantic relationships and causing candidate redundancy. Meanwhile, current geometry-aware methods often require complex optimization or directly reweighting the original token probabilities, leading to significant computational overhead or inference instability. To address this, we formulate decoding as a subset optimization problem using a Mahalanobis distance-driven objective to enhance semantic diversity while preserving high probabilities. Specifically, we dynamically discount redundant generation paths using a token similarity matrix, constructed via an adaptive-bandwidth kernel over token embeddings. We further devise an efficient greedy selection algorithm with near-linear complexity in the candidate size under early stopping, while establishing its theoretical approximation guarantees. This renders ME-Decoding a robust, plug-and-play module with negligible inference overhead. Extensive experiments across diverse reasoning and generation tasks demonstrate that our method consistently achieves strong performance.

Dunyao Xue, Chengshuo Du, Zhengbo Wang +2

Institute of Statistics and Big Data, Renmin University of China, Beijing, China · Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing, China · Center for Applied Statistics, Institute of Statistics and Big Data, Renmin University of China, Beijing, China

Language Model Decoding Diverse Text Generation

Ask PDF