SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models

Ziwei Li; Yuang Ma; Yi Kang

cs.LGApr 6, 2026

SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models

Authors: Ziwei Li, Yuang Ma, Yi Kang

Organizations: School of Microelectronics, University of Science and Technology of China, Hefei, Anhui, China

Abstract

The rapid growth of large language models (LLMs) presents significant deployment challenges due to their massive computational and memory demands. While model compression, such as network pruning, offers potential solutions, most existing methods often fail to maintain good performance at high compression ratios. To address this, we propose SLaB, a novel framework that decomposes each linear layer weight into three complementary components: a sparse matrix, a low-rank matrix, and a binary matrix. SLaB eliminates the need for retraining and leverages activation-aware pruning scores to guide the decomposition process. Experiments on Llama-family models demonstrate that SLaB achieves state-of-the-art performance, reducing perplexity by up to 36% compared to existing methods at 50% compression and improving accuracy by up to 8.98% over the baseline on zero-shot tasks.

Figures & tables

Fig. 1: Compression of the Llama-2 7B model [ 16 ] using only low-rank and sparse matrices: perplexity comparison on the WikiText-2 dataset under different rank settings at a

50\%

compression ratio.

Fig. 2: Overview of the SLaB framework.

Fig. 3: Variation of the average Frobenius norm difference between compressed and original layers with respect to rank. Experiments are conducted on the Llama-2 7B model [ 16 ] with a

50\%

compression ratio.

Method	Sparsity( CR )	Llama-3.2 1B		Llama-2 7B		Llama-3 8B
Method	Sparsity( CR )	ppl 2 $\downarrow$	acc 2 $\uparrow$	ppl $\downarrow$	acc $\uparrow$	ppl $\downarrow$	acc $\uparrow$
Dense	$0\%$	$9.06$	$58.3$	$5.12$	$68.4$	$5.75$	$72.9$
SparseGPT	US 1 ( $50\%$ )	$18.09$	$51.2$	$6.52$	$63.9$	$8.73$	$66.9$
Wanda	US ( $50\%$ )	$21.37$	$48.5$	$6.45$	$64.0$	$9.15$	$65.4$
SLaB	US ( $50\%$ )	$\boldsymbol{11.57}$	$\boldsymbol{55.8}$	$\boldsymbol{5.49}$	$\boldsymbol{66.2}$	$\boldsymbol{6.67}$	$\boldsymbol{70.7}$
SparseGPT	4:8 ( $50\%$ )	$21.64$	$48.0$	$7.94$	$61.6$	$11.33$	$61.3$

TABLE I: Comparison of perplexity on WikiText-2 and average zero-shot accuracies (

\%

) at different compression rates.

Comparison	$\left(1,\displaystyle\frac{D_{\text{in}}}{32}\right)$	$\left(1,\displaystyle\frac{D_{\text{in}}}{16}\right)$	$(1,D_{\text{in}})$	$(16,D_{\text{in}})$	$(32,D_{\text{in}})$
Group	$\left(1,\displaystyle\frac{D_{\text{in}}}{32}\right)$	$\left(1,\displaystyle\frac{D_{\text{in}}}{16}\right)$	$(1,D_{\text{in}})$	$(16,D_{\text{in}})$	$(32,D_{\text{in}})$
ppl $\downarrow$	$5.516$	$5.491$	$5.493$	$5.544$	$5.546$
acc $\uparrow$ ( $\%$ )	$65.6$	$65.8$	$66.2$	$65.9$	$66.2$
Iterations	1	10	20	30	40
ppl( $\downarrow$ )	$5.678$	$5.531$	$5.493$	$5.480$	$5.477$

TABLE II: Hyperparameter exploration with Llama-2 7B (

\text{CR}=50\%

) model.

Accuracy ( $\%$ )	ARC-C	ARC-E	RTE	WinoGrande	Avg
$\mathbf{W}_{\text{S}}$	$32.0$	$58.2$	$53.1$	$62.0$	$49.8$
$\mathbf{W}_{\text{S}}+\mathbf{W}_{\text{L}}(r=16)$	$33.4$	$59.2$	$53.8$	$64.8$	$51.2$
$\mathbf{W}_{\text{S}}+\text{factor}{\textsuperscript{*}}\odot\mathbf{W}_{\text{B}}$	$42.1$	$71.8$	$55.6$	$68.2$	$58.2$
$\mathbf{W}_{\text{S}}+\mathbf{W}_{\text{L}}\odot\mathbf{W}_{\text{B}}$	$43.2$	$71.3$	$57.0$	$68.6$	$58.9$

TABLE III: Ablation study with Llama-2 7B (2:4,

\text{CR}=50\%

) model.

Binary Quantization

See all 9 papers

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Mar 12, 2026Xinhao Huang, You-Liang Huang, Zeyi WenLLM Compression Low-Rank Compression
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
Mar 2, 2026Mehdi Makni, Xiang Meng, Rahul MazumderLLM Compression Low-Rank Compression
Operations Research Center Massachusetts Institute of Technology
SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

Mar 12, 2026J/K move · Enter open · S save

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Xinhao Huang, You-Liang Huang, Zeyi Wen

The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

LLM Compression Low-Rank Compression Activation Sparsity

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but the billion-scale parameters pose deployment challenges. Although existing methods attempt to reduce the scale of LLMs, they require either special hardware support or expensive post-training to maintain model quality. To facilitate efficient and affordable model slimming, we propose a novel training-free compression method for LLMs, named "SoLA", which leverages Soft activation sparsity and Low-rAnk decomposition. SoLA can identify and retain a minority of components significantly contributing to inference, while compressing the majority through low-rank decomposition, based on our analysis of the activation pattern in the feed-forward network (FFN) of modern LLMs. To alleviate the decomposition loss, SoLA is equipped with an adaptive component-wise low-rank allocation strategy to assign appropriate truncation positions for different weight matrices. We conduct extensive experiments on LLaMA-2-7B/13B/70B and Mistral-7B models across a variety of benchmarks. SoLA exhibits remarkable improvement in both language modeling and downstream task accuracy without post-training. For example, with a 30% compression rate on the LLaMA-2-70B model, SoLA surpasses the state-of-the-art method by reducing perplexity from 6.95 to 4.44 and enhancing downstream task accuracy by 10%.

Abstract

Binary Quantization

Explore similar work

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs

SPQ: An Ensemble Technique for Large Language Model Compression