SVD Contextual Sparsity Predictors for Fast LLM Inference

Georgii Serbin; Kirill Koshkin; Zhongao Sun; Anastasiya Bistrigova; C. C. Korikov

cs.LGMar 14, 2026

SVD Contextual Sparsity Predictors for Fast LLM Inference

Authors: Georgii Serbin, Kirill Koshkin, Zhongao Sun, Anastasiya Bistrigova, C. C. Korikov

Organizations: Huawei Technologies Ltd. · Moscow State University, Russia.

Abstract

Contextual sparsity is one of the approaches used to reduce computational complexity in the inference process of large language models (LLMs). Existing techniques for efficient LLM inference acceleration based on contextual sparsity with minimal accuracy degradation require training sparse pattern predictors. This paper presents a framework for accelerating inference of ReGLU-based feed-forward networks (FFNs) within LLMs. The proposed framework provides a fast, training-free method for building sparse pattern predictors using truncation-aware singular value decomposition (SVD) of the gate projection matrix, along with a threshold calibration algorithm, and inference executors supporting conditional computation on CUDA and CANN devices. Experiments on three sparse LLMs with an average activation sparsity level of 90% in the FFNs demonstrate up to a 1.8x reduction in end-to-end decoding time while maintaining less than 1% degradation in benchmark scores on tasks involving complex math and code generation. This work advances the deployment of LLMs on edge devices.

Figures & tables

Figure 1: Overview of our framework. During the offline stage, sparsity predictors are constructed using low-rank SVD approximation with calibrated per-neuron thresholds. At inference time, SVD-based predictors precede each FFN block to exploit contextual sparsity and accelerate generation.

Figure 2: Distribution of SVD-based predictor outputs for a sample neuron. Although active and inactive states are well separated, zero thresholding causes significant false negatives for truly active neurons.

Figure 3: Sparse inference using SVD-based sparsity predictors. Neurons with positive predictor values are considered active during gate projection. Gate and up projections are executed sequentially to increase sparsity.

Model	Method	$s$	Accuracy, %	Speedup
ProSparse-LLaMA2-7B	Dense	$-$	38.92	1.00 $\times$
	SVDP	0.4	38.66 (-0.26)	1.61 $\times$
		0.5	38.78 (-0.14)	1.64 $\times$
		0.6	38.77 (-0.15)	1.67 $\times$
		0.7	38.57 (-0.35)	1.71 $\times$
TurboSparse-Mistral-Instruct	Dense	$-$	54.30	1.00 $\times$

Table 1: Average accuracy when varying hyperparameter

s

for different models.

s

implicitly sets sparsity, which directly translates to end-to-end speedup.

Method	Predicted sparsity, %	TriviaQA	ARC-E	ARC-C	HumanEval	MBPP	GSM8K	BBH	CMMLU	Average	E2E
	Predicted sparsity, %	acc, %	acc, %	acc, %	pass@1, %	pass@1, %	acc, %	acc, %	acc, %	Score	Speedup
Dense baseline	$-$	62.94	75.42	53.58	17.68	22.59	11.45	35.85	31.84	38.92	1.00 $\times$
Trained predictors
Deja Vu	20	59.47	74.87	53.41	3.66	2.98	3.03	34.49	30.97	32.86	1.38 $\times$
PowerInfer	80	61.10	74.54	52.99	15.85	21.15	10.69	35.29	31.80	37.93	1.63 $\times$
Training-free methods

Table 2: Comparison with other methods on ProSparse-LLaMA2-7B, evaluated in our framework (

r=1024

parallel,

r=256

sequential).

Figure 4: Layer-wise ROC AUC score of different predictors averaged across several sequence samples for ProSparse-LLaMA2-7B. AUROC score of 1 indicates perfect separation and AUROC score of 0.5 corresponds to random guessing.

Configuration	ProSparseLLAMA2-7B			TurboSparse-Mistral-Instruct			SparseQwen2-7B
	GSM8K	HumanEval	$\text{Average}^{\dagger}$	GSM8K	HumanEval	$\text{Average}^{\dagger}$	GSM8K	HumanEval	$\text{Average}^{\dagger}$
Dense baseline	11.45	17.68	38.92	61.71	34.15	54.30	77.1	64.02	65.84
Parallel pipeline
Naive SVD	6.60	10.98	34.08	60.20	24.39	51.52	74.00	53.66	62.05
+ Data whitening	10.16	16.46	37.10	60.88	37.20	53.02	75.89	61.59	63.99
+ Bias calibration	10.54	17.68	38.72	62.24	36.59	54.34	76.65	62.20	65.37

Table 3: SVD sparsity estimator ablation study. † Averaged across all benchmarks (see Appendix D ).

Figure 5: Layer-wise ROC-AUC: data-aware SVD vs. Naive SVD for ProSparse-LLaMA2-7B.

LLM architecture	Sparsity, %	FFN Speedup
LLM architecture	Sparsity, %	CUDA	CANN
ProSparseLLAMA2-7B	20	1.30 $\times$	0.76 $\times$
	50	1.90 $\times$	1.23 $\times$
	80	3.34 $\times$	3.38 $\times$
	95	4.67 $\times$	30.67 $\times$
TurboSparse-Mistral-Instruct	20	1.29 $\times$	0.76 $\times$

Table 4: FFN performance of our CUDA and CANN executors at different sparsity ratios. The speedups are compared to the corresponding dense implementations. Predictors latency is not included. For this experiment, sparsity ratio is fixed across all linear layers.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Mean Kendall

\tau_{i}^{\mathrm{K}}

across neurons for proxy-sorted

\eta

-step sums from ProSparse-LLaMA2-7B. The x-axis shows layer number. For clarity, layer-wise sparsity ratios are depicted. The shaded regions denote the 10th to 90th percentile interval, illustrating the central 80% of the data distribution.

Method	TriviaQA	ARC-C	ARC-E	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average
	acc, %	acc, %	acc, %	pass@1, %	pass@1, %	acc, %	acc, %	acc, %
Dense mode	60.96	86.43	93.81	60.98	44.46	36.85	47.81	79.58	63.86
SVDP	51.48	86.43	93.81	49.39	36.55	37.83	47.61	79.58	60.34

Appendix

Table 5: Qwen2-7B-Instruct accuracy results, 25% average predicted sparsity

Method	$s$	TriviaQA	ARC-E	ARC-C	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average	E2E
		acc, %	acc, %	acc, %	pass@1, %	pass@1, %	acc, %	acc, %	acc, %	Score	Speedup
Dense mode	-	62.94	75.42	53.58	17.68	22.59	11.45	35.85	31.84	38.92	1.00 $\times$
SVDP	0.4	62.38	75.29	53.67	17.68	22.38	10.61	35.46	31.81	38.66	1.61 $\times$
	0.5	62.74	75.34	53.67	17.68	22.07	11.22	35.58	31.94	38.78	1.64 $\times$
	0.6	62.90	75.29	53.75	18.29	21.97	10.54	35.54	31.91	38.77	1.67 $\times$
	0.7	62.04	75.42	53.58	17.07	21.87	11.22	35.43	31.90	38.57	1.71 $\times$

Appendix

Table 6: ProSparse-LLaMA2-7B evaluation results,

r=256

Method	$s$	TriviaQA	ARC-E	ARC-C	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average	E2E
		acc, %	acc, %	acc, %	pass@1, %	pass@1, %	acc, %	acc, %	acc, %	Score	Speedup
Dense mode	-	48.58	87.21	71.67	34.15	42.20	61.71	42.35	46.52	54.30	1.00 $\times$
SVDP	0.4	45.46	87.08	71.67	32.93	42.09	61.49	42.46	46.55	53.72	1.67 $\times$
	0.5	43.26	87.04	71.76	33.54	42.71	61.41	42.32	46.47	53.56	1.73 $\times$
	0.6	40.63	87.08	72.10	33.54	42.09	61.41	42.19	46.43	53.18	1.79 $\times$
	0.7	39.21	87.21	72.10	37.20	42.09	60.96	42.33	46.45	53.44	1.84 $\times$

Appendix

Table 7: TurboSparse-Mistral-Instruct evaluation results,

r=352

Method	$s$	TriviaQA	ARC-E	ARC-C	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average	E2E
		acc, %	acc, %	acc, %	pass@1, %	pass@1, %	acc, %	acc, %	acc, %	Score	Speedup
Dense mode	-	50.76	91.92	83.87	64.02	46.3	77.1	48.42	64.35	65.84	1.00 $\times$
SVDP	0.4	50.87	91.92	83.87	62.80	44.56	76.50	48.40	64.35	65.41	1.78 $\times$
	0.5	49.96	91.92	83.87	62.80	43.53	76.80	48.40	64.35	65.20	1.81 $\times$
	0.6	48.82	91.92	83.87	63.41	43.53	76.80	48.34	64.35	65.13	1.85 $\times$
	0.7	47.42	91.92	83.87	62.20	41.99	75.82	48.36	64.35	64.49	1.86 $\times$

Appendix

Table 8: SparseQwen2-7B evaluation results,

r=512

	TriviaQA	ARC-E	ARC-C	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average
Dense baseline	62.94	75.42	53.58	17.68	22.59	11.45	35.85	31.84	38.92
Parallel pipeline
Naive SVD	58.19	69.91	49.32	10.98	13.35	6.60	33.98	30.28	34.08
+ Data whitening	53.85	75.34	53.75	16.46	19.92	10.16	35.46	31.83	37.10
+ Bias calibration	62.66	75.38	53.58	17.68	22.18	10.54	35.86	31.85	38.72
Sequential pipeline

Appendix

Table 9: SVD sparsity estimator ablation study for ProSparseLLAMA2-7B.

	TriviaQA	ARC-E	ARC-C	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average
Dense baseline	50.76	91.92	83.87	64.02	46.30	77.10	48.42	64.35	65.84
Parallel pipeline
Naive SVD	46.24	91.92	83.87	53.66	34.39	74.00	48.00	64.35	62.05
+ Data whitening	42.92	91.92	83.87	61.59	43.02	75.89	48.37	64.35	63.99
+ Bias calibration	50.20	91.92	83.87	62.20	45.28	76.65	48.45	64.35	65.37
Sequential pipeline

Appendix

Table 10: SVD sparsity estimator ablation study for SparseQwen2-7B.

	TriviaQA	ARC-E	ARC-C	Humaneval	MBPP	GSM8K	BBH	CMMLU	Average
Dense baseline	48.58	87.21	71.67	34.15	42.20	61.71	42.35	46.52	54.30
Parallel pipeline
Naive SVD	41.40	87.04	71.76	24.39	38.91	60.20	42.13	46.34	51.52
+ Data whitening	37.98	86.57	71.25	37.20	41.79	60.88	42.27	46.18	53.02
+ Bias calibration	46.21	87.12	71.76	36.59	41.89	62.24	42.44	46.45	54.34
Sequential pipeline

Appendix

Table 11: SVD sparsity estimator ablation study for TurboSparse-Mistral-Instruct.

	hyperparameters
Model	hidden size	intermediate size	n attention heads	n KV heads
ProSparse-LLaMA2-7B	4096	11008	32	32
TurboSparse-Mistral-Instruct	4096	14336	32	8
SparseQwen2-7B	3584	18944	28	4

Appendix

Table 12: LLMs hyperparameter values.

Figure 7: ROC AUC scores of SVD-predictors at different layers averaged across several sequence samples for ProSparse-LLaMA2-7B. The brighter curves represent deeper layers. For most of the layers a quality metric reaches a plateau at relatively small rank.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference Acceleration Long-Context Language Model Inference
Department of Computer Science, Rutgers University · Oceanbase, Ant Group.
Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Activation Sparsity Structured Sparsity
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications
Inference Time Context Sparsity: Illusion or Opportunity?
May 22, 2026Sahil Joshi, Prithvi Dixit, Agniva Chowdhury +5LLM Inference Efficiency Structured Sparsity
α Rice University · β UC Berkeley · γ IIT Bombay

Mar 30, 2026J/K move · Enter open · S save

CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

Chuxu Song, Zhencan Peng, Jiuqi Wei, Chuanhui Yang

Department of Computer Science, Rutgers University · Oceanbase, Ant Group.

LLM Inference Acceleration Long-Context Language Model Inference Attention Mechanisms Sparse Attention

Open paper AskPDF arXiv

Long-context LLMs increasingly rely on extended, reusable prefill prompts for agents and domain Q&A, pushing attention and KV-cache to become the dominant decode-time bottlenecks. While sparse attention reduces computation and transfer costs, it often struggles to maintain accuracy at high sparsity levels due to the inherent distribution shift between Queries and Keys. We propose Centroid-Scoring Attention (CSAttention), a training-free sparse attention method optimized for high-throughput serving of reusable contexts. CSAttention adopts a storage-for-computation strategy tailored to the offline-prefill/online-decode setting: it front-loads computation into a one-time offline prefill phase that can be amortized across multiple queries, while aggressively optimizing per-step decoding latency. Specifically, CSAttention constructs query-centric lookup tables during offline prefill, whose size remains fixed during decoding, and enables online decoding to replace full-context scans with efficient table lookups and GPU-friendly score accumulation. Extensive experiments demonstrate that CSAttention achieves near-identical accuracy to full attention. Under high sparsity (95%) and long-context settings (32K-128K), CSAttention consistently outperforms state-of-the-art sparse attention methods in both model accuracy and inference speed, achieving up to 4.6x inference speedup over the most accurate baseline at a context length of 128K.

Abstract

LLM Inference Acceleration

Explore similar work

CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

Inference Time Context Sparsity: Illusion or Opportunity?