SPQ: An Ensemble Technique for Large Language Model Compression

Jiamin Yao; Eren Gultepe

cs.CLFeb 20, 2026

SPQ: An Ensemble Technique for Large Language Model Compression

Authors: Jiamin Yao, Eren Gultepe

Organizations: Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

Abstract

This study presents an ensemble technique, SPQ (SVD-Pruning-Quantization), for large language model (LLM) compression that combines variance-retained singular value decomposition (SVD), activation-based pruning, and post-training linear quantization. Each component targets a different source of inefficiency: i) pruning removes redundant neurons in MLP layers, ii) SVD reduces attention projections into compact low-rank factors, iii) and 8-bit quantization uniformly compresses all linear layers. At matched compression ratios, SPQ outperforms individual methods (SVD-only, pruning-only, or quantization-only) in perplexity, demonstrating the benefit of combining complementary techniques. Applied to LLaMA-2-7B, SPQ achieves up to 75% memory reduction while maintaining or improving perplexity (e.g., WikiText-2 5.47 to 4.91) and preserving accuracy on downstream benchmarks such as C4, TruthfulQA, and GSM8K. Compared to strong baselines like GPTQ and SparseGPT, SPQ offers competitive perplexity and accuracy while using less memory (6.86 GB vs. 7.16 GB for GPTQ). Moreover, SPQ improves inference throughput over GPTQ, achieving up to a 1.9x speedup, which further enhances its practicality for real-world deployment. The effectiveness of SPQ's robust compression through layer-aware and complementary compression techniques may provide practical deployment of LLMs in memory-constrained environments. Code is available at: https://github.com/JiaminYao/SPQ_LLM_Compression/

Figures & tables

Figure 1: SPQ Framework. 1) Attention layers are compressed using variance-retained SVD, where the preserved rank is determined by a variance threshold. 2) MLP layers, per-layer activation means are computed and mapped to pruning ratios via a log-scale formula. 3) Finally, 8-bit linear quantization with different scaling strategies is applied to all linear layers.

Quant Mode	Scale $s_{i}$
Per-tensor	$s_{i}$ (one scale per layer)
Per-channel	$s_{j}$ (one scale per output channel)
LNH	$s_{j}$ if $l\in\text{Attention}$ else $s_{i}$
PBH	$s_{j}$ if $q^{(l)}\geq P_{\alpha}$ else $s_{i}$
MSH	$s_{j}$ if $q^{(l)}\geq\mu+k\sigma$ else $s_{i}$

Table 1: Quantization modes scale selection rules.

Figure 2: Per-layer SVD performance.

Figure 3: Per-layer structural pruning performance.

Figure 4: Pruning performance with different per-layer ratios and methods.

Figure 5: 8-bit linear quantization performance.

Figure 6: Comparison of ensemble compression methods. Top row: Perplexity. Bottom row: Throughput.

Variant	MEM (GB)	Perplexity
Q	7.21 (–, –)	4.72 (–, –)
S+Q	7.10 ( $t=$ -8.25, $p<$ .001)	4.76 ( $t=$ 1.82, $p=$ .954)
P+Q	7.05 ( $t=$ -8.45, $p<$ .001)	4.82 ( $t=$ 2.29, $p=$ .980)

Table 2: Weight Memory(MEM): one-sample one-sided t-test vs. constant Quant-only. Perplexity: two-sample one-sided t-test.

RATIO (MEM.)	METHOD	WikiText-2 $\downarrow$	C4 $\downarrow$	Openb.	ARC-e	WinoG.	HellaS.	PIQA	Average $\uparrow$	TruthfulQA-1 $\uparrow$	TruthfulQA-2 $\uparrow$	GSM8K $\uparrow$
0% (26.95 GB)	Original	5.47	6.85	0.31	0.75	0.70	0.51	0.78	0.61	0.24	0.39	0.05
21% (21.41 GB)	ASVD	6.54	7.66	0.32	0.70	0.66	0.50	0.76	0.59	0.25	0.41	0.03
50% (13.40 GB)	SparseGPT	7.76	8.98	0.27	0.63	0.65	0.47	0.75	0.56	0.22	0.38	0.03
73% (7.16 GB)	GPTQ	5.48	6.66	0.34	0.74	0.70	0.51	0.79	0.62	0.22	0.34	0.04
75% (6.86 GB)	SPQ	4.91	7.11	0.30	0.72	0.68	0.51	0.77	0.60	0.24	0.38	0.05

Table 3: Performance of LLaMA-2-7B compressed by SPQ and baselines on: i) 2 language modeling datasets with perplexity (

\downarrow

), WikiText-2 ( Thoma et al., 2025 ) and C4 ( Raffel et al., 2020 ) ; ii) 5 reasoning datasets with individual and average accuracy (

\uparrow

), OpenBookQA ( Mihaylov et al., 2018 ) , ARC ( Clark et al., 2018 ) , WinoGrande ( Sakaguchi et al., 2021 ) , HellaSwag ( Zellers et al., 2019 ) , PIQA ( Bisk et al., 2020 ) ; 2 reasoning datasets with BLEU (

\uparrow

), TruthfulQA-1 and TruthfulQA-2 ( Lin et al., 2021 ) ; and 1 reasoning dataset with Exact Match Flexible-Extract Accuracy (

\uparrow

) GSM8K ( Cobbe et al., 2021 ) .

Figure 7: SPQ and individual method perplexities.

Model	MEM (GB)			Perplexity
	Before	After	Ratio	Before	After	Change
LLaMA-3.2-1B	5.99	2.27	62%	7.88	8.62	+0.75
LLaMA-3.2-3B	14.42	4.77	67%	6.86	6.39	-0.47
LLaMA-2-7B	26.95	7.05	74%	5.47	4.71	-0.76
OPT-1.3B	5.68	1.72	70%	14.80	12.76	-2.04
OPT-2.7B	11.12	3.11	72%	12.81	11.49	-1.31

Table 4: WikiText-2 perplexity and weight memory of LLaMA, OPT, Vicuna, and Mistral models before and after applying SPQ (MLP pruning with max ratio 0.05, attention SVD with 0.94 variance threshold, and 8-bit linear quantization).

Figure 8: SPQ and GPTQ throughput comparisons.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 272 papers

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Mar 12, 2026Xinhao Huang, You-Liang Huang, Zeyi WenLLM Compression Low-Rank Compression
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
Aug 25, 2026Mohammad MozaffariLLM Compression Neural Network Compression
Department of Computer Science University of Toronto
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis

Mar 12, 2026J/K move · Enter open · S save

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Xinhao Huang, You-Liang Huang, Zeyi Wen

The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

LLM Compression Low-Rank Compression Activation Sparsity

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but the billion-scale parameters pose deployment challenges. Although existing methods attempt to reduce the scale of LLMs, they require either special hardware support or expensive post-training to maintain model quality. To facilitate efficient and affordable model slimming, we propose a novel training-free compression method for LLMs, named "SoLA", which leverages Soft activation sparsity and Low-rAnk decomposition. SoLA can identify and retain a minority of components significantly contributing to inference, while compressing the majority through low-rank decomposition, based on our analysis of the activation pattern in the feed-forward network (FFN) of modern LLMs. To alleviate the decomposition loss, SoLA is equipped with an adaptive component-wise low-rank allocation strategy to assign appropriate truncation positions for different weight matrices. We conduct extensive experiments on LLaMA-2-7B/13B/70B and Mistral-7B models across a variety of benchmarks. SoLA exhibits remarkable improvement in both language modeling and downstream task accuracy without post-training. For example, with a 30% compression rate on the LLaMA-2-70B model, SoLA surpasses the state-of-the-art method by reducing perplexity from 6.95 to 4.44 and enhancing downstream task accuracy by 10%.

Abstract

LLM Compression

Explore similar work

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression