SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Xinhao Huang; You-Liang Huang; Zeyi Wen

cs.CLMar 12, 2026

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Authors: Xinhao Huang, You-Liang Huang, Zeyi Wen

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Abstract

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but the billion-scale parameters pose deployment challenges. Although existing methods attempt to reduce the scale of LLMs, they require either special hardware support or expensive post-training to maintain model quality. To facilitate efficient and affordable model slimming, we propose a novel training-free compression method for LLMs, named "SoLA", which leverages Soft activation sparsity and Low-rAnk decomposition. SoLA can identify and retain a minority of components significantly contributing to inference, while compressing the majority through low-rank decomposition, based on our analysis of the activation pattern in the feed-forward network (FFN) of modern LLMs. To alleviate the decomposition loss, SoLA is equipped with an adaptive component-wise low-rank allocation strategy to assign appropriate truncation positions for different weight matrices. We conduct extensive experiments on LLaMA-2-7B/13B/70B and Mistral-7B models across a variety of benchmarks. SoLA exhibits remarkable improvement in both language modeling and downstream task accuracy without post-training. For example, with a 30% compression rate on the LLaMA-2-70B model, SoLA surpasses the state-of-the-art method by reducing perplexity from 6.95 to 4.44 and enhancing downstream task accuracy by 10%.

Figures & tables

Figure 1: Framework of the proposed SoLA. We initially recognize the soft activation sparsity within the feed-forward network. Leveraging this property, we introduce a fine-grained model decomposition technique to preserve model quality. Furthermore, to alleviate the compression error of SVD, we develop an adaptive component-wise truncation strategy to allocate appropriate truncation positions for different types of weight matrices.

Figure 2: Accumulation of

\delimiter 69645069\mathchar 29016\mathchar 29015\delimiter 69645069_{\mathchar 28998}^{\mathchar 28722}

and distribution of

\delimiter 69645069\mathchar 29016\mathchar 29015\delimiter 69645069_{\mathchar 28998}

across neurons in different layers of LLaMA-2-7B and LLaMA-2-13B on WikiText2 and c4 datasets, sorted from largest to smallest, highlighting the soft activation sparsity phenomenon.

LLaMA- 2-13B	original	PN	MN
LLaMA- 2-13B	original	1%	10%	30%	50%
perplexity ( $\delimiter 52573049$ )	4.57	9665.4	4.83	6.58	17.03

Table 1: Impact of neuron pruning on LLaMA-2-13B model perplexity, highlighting the sensitivity to the loss of high-norm “Prime Neurons” (PN) and the resilience following the removal of low-norm “Marginal Neurons” (MN).

Methods	Ratio	Average	MMLU	BoolQ	PIQA	WinoGrande	HellaSwag	ARC-e	ARC-c	OBQA
LLaMA-2-7B	0%	0.6410	0.457	0.7777	0.7905	0.6938	0.7592	0.7449	0.4625	0.442
LLM-Pruner	20%	0.5512	0.262	0.6376	0.7595	0.6338	0.6783	0.6431	0.3993	0.396
FLAP		0.5318	0.319	0.5394	0.7454	0.6298	0.6474	0.6128	0.3643	0.396
SliceGPT		0.4184	0.263	0.3792	0.6126	0.5983	0.4428	0.4609	0.2841	0.306
Bolaco		0.5733	0.343	0.7201	0.7509	0.6561	0.6433	0.6819	0.3748	0.416
SVD-LLM		0.4673	0.268	0.5468	0.6513	0.6243	0.5173	0.4722	0.2782	0.380

Table 2: Downstream task accuracy of the compressed LLaMA-2-7B/13B/70B models. Bold denotes the best result at the same compression ratio, while underline indicates the second best result.

Method	Ratio	LLaMA-2			Mistral
Method	Ratio	7B	13B	70B	7B
Dense	0 $\%$	5.11	4.57	3.12	4.92
LLM-Pruner	20 $\%$	10.55	9.67	-	-
FLAP		6.76	5.90	8.76	7.11
SliceGPT		9.70	8.21	5.76	8.23
Bolaco		7.31	6.34	-	-

Table 3: WikiText2 validation perplexity of pruning methods for LLaMA-2 model series and Mistral-7B-v0.1.

Figure 3: Perplexity of WikiText2 among different methods on LLaMA-2-13B.

Figure 4: The impact of “Prime Neurons” ratios on LLaMA-2-13B perplexity under 20% and 30% compression ratios.

Model	Ratio	Perplexity ( $\delimiter 52573049$ )		Avg. Acc. ( $\delimiter 52568952$ )
Model	Ratio	Unif.	Adap.	Unif.	Adap.
LLaMA-2-7B	20 $\%$	8.07	7.18	0.467	0.541
LLaMA-2-13B		6.18	6.52	0.557	0.564
Mistral-7B		7.26	6.68	0.528	0.578
LLaMA-2-7B	30 $\%$	11.40	9.32	0.425	0.492
LLaMA-2-13B		7.93	7.02	0.485	0.541

Table 4: Comparison of perplexity and average accuracy of downstream tasks between uniform and adaptive strategy.

Figure 5: Perplexity of LLaMA-2-13B under 30% compression ratio using calibration data with different numbers (32, 64, 128, 256) and types (WikiText2 and C4).

Ratio	Total Time of Operation (speedup)
Ratio	7B	13B	70B
0%	20.04	31.64	96.58
20%	16.39 (1.22 $\mathchar 8706$ )	21.92 (1.44 $\mathchar 8706$ )	65.76 (1.47 $\mathchar 8706$ )
30%	13.04 (1.54 $\mathchar 8706$ )	17.87 (1.77 $\mathchar 8706$ )	57.04 (1.69 $\mathchar 8706$ )

Table 5: Operation cost of LLaMA-2 series.

Methods	Ratio	Average	MMLU	BoolQ	PIQA	WinoGrande	HellaSwag	ARC-e	ARC-c	OBQA
Mistral-7B	0%	0.701	0.625	0.8398	0.8205	0.7395	0.8102	0.7955	0.5392	0.44
FLAP	20%	0.500	0.259	0.6226	0.7231	0.6409	0.5594	0.5105	0.3191	0.368
SliceGPT		0.427	0.286	0.3786	0.6066	0.5943	0.4510	0.4815	0.3003	0.320
SVD-LLM		0.578	0.418	0.6829	0.7339	0.6843	0.6175	0.7134	0.4053	0.366
SoLA (Ours)		0.581	0.442	0.6609	0.7367	0.6875	0.6332	0.6999	0.3976	0.392
FLAP	30%	0.496	0.264	0.6526	0.6959	0.6480	0.5561	0.4891	0.3055	0.358

Table 6: Downstream task accuracy of the compressed Mistral-7B models. Bold denotes the best result at the same compression ratio, while underline indicates the second best result.

Model	Compression Ratio	Operations( $\mathchar 29037\mathchar 29043$ )
Model	Compression Ratio	Gate	Up	Down	Q	K	O	Total (speedup)
LLaMA-2-7B	Dense	4.94	4.79	5.09	1.75	1.74	1.74	20.04
	20%	2.92	3.52	6.66	0.87	0.69	1.74	16.39 (1.22 $\mathchar 8706$ )
	30%	2.92	3.00	4.48	0.69	0.69	1.27	13.04 (1.54 $\mathchar 8706$ )
LLaMA-2-13B	Dense	7.14	7.12	8.11	3.07	3.09	3.12	31.64
	20%	5.61	5.08	5.44	1.34	1.34	3.12	21.92 (1.44 $\mathchar 8706$ )

Table 7: Results of timing the matrix multiplications in each component of LLaMA-2 series.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 272 papers

SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models
Apr 6, 2026Ziwei Li, Yuang Ma, Yi KangBinary Quantization LLM Compression
School of Microelectronics, University of Science and Technology of China, Hefei, Anhui, China
Optimizing Singular Spectrum for Large Language Model Compression
Feb 20, 2025Dengjie Li, Tiancheng Shen, Yao Zhou +3Model Compression LLM Compression
SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

Apr 6, 2026J/K move · Enter open · S save

SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models

Ziwei Li, Yuang Ma, Yi Kang

School of Microelectronics, University of Science and Technology of China, Hefei, Anhui, China

Binary Quantization LLM Compression LLM Pruning Low-Rank Compression

Open paper AskPDF arXiv

The rapid growth of large language models (LLMs) presents significant deployment challenges due to their massive computational and memory demands. While model compression, such as network pruning, offers potential solutions, most existing methods often fail to maintain good performance at high compression ratios. To address this, we propose SLaB, a novel framework that decomposes each linear layer weight into three complementary components: a sparse matrix, a low-rank matrix, and a binary matrix. SLaB eliminates the need for retraining and leverages activation-aware pruning scores to guide the decomposition process. Experiments on Llama-family models demonstrate that SLaB achieves state-of-the-art performance, reducing perplexity by up to 36% compared to existing methods at 50% compression and improving accuracy by up to 8.98% over the baseline on zero-shot tasks.

Abstract

LLM Compression

Explore similar work

SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models

Optimizing Singular Spectrum for Large Language Model Compression

SPQ: An Ensemble Technique for Large Language Model Compression