Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Peng Liu; Huibing Zeng; Yiqun Zhang; Yang Yi; Jigang Wu

cs.LGAug 25, 2026

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Authors: Peng Liu, Huibing Zeng, Yiqun Zhang, Yang Yi, Jigang Wu

Organizations: School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Information and Artificial Intelligence, Yangzhou University, Yangzhou 225000, China

Abstract

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.

Figures & tables

Fig. 1 : Comparison of pruning architecture. (a) pre-finetune pruning; (b) post-finetune pruning; (c) REP-LIE.

Fig. 2 : The schematics of LoRA on a single layer.

Fig. 3 : Overall framework diagram of REP-LIE. (a) Employing the gradients of LoRA’s low-rank matrices to estimate importance; (b) Stability-Aware progressive structured pruning.

Method	#Param	GLUE benchmark						Wilcoxon Sig. vs REP-LIE
Method	#Param	QQP	QNLI	SST-2	CoLA	MRPC	RTE	Wilcoxon Sig. vs REP-LIE
BERT-base	109M	91.50	91.40	93.23	58.90	88.48	66.80	$p>0.05$
CoFi	67M	90.70	89.10	91.20	52.40	87.30	64.60	pass ( $p=0.0156$ )
PGB	67M	91.10	90.21	92.27	55.01	84.26	64.25	pass ( $p=0.0156$ )
DynaBERT	67M	90.68	88.23	91.47	51.18	77.90	63.21	pass ( $p=0.0156$ )
EBERT	67M	90.49	89.74	90.68	N.A.	72.08	52.75	pass ( $p=0.0312$ )

TABLE I: Model compression results with 50% sparsity.

Sparsity	Method	#Params	PPL $\downarrow$	BoolQ	PIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Avg.	Wilcoxon Sig. vs REP-LIE
0%	LLaMA-7B	6.7B	12.63	75.08	79.16	76.20	70.00	72.89	44.88	44.40	66.09	–
20%	LLM-Pruner	5.4B	18.01	66.76	78.45	71.44	63.77	66.41	39.85	43.80	61.50	pass ( $p=0.0078$ )
	Compresso		–	79.08	75.46	53.44	67.80	68.64	37.97	34.20	59.51	pass ( $p=0.0156$ )
	SlimGPT		16.68	74.59	78.94	74.40	68.43	70.50	43.26	45.40	65.07	pass ( $p=0.0156$ )
	REP-LIE(ours)		16.32	78.78	79.91	75.12	69.14	75.93	43.17	45.52	66.80	–
25%	LLM-Pruner	5.0B	20.57	62.81	76.93	69.21	60.46	63.34	38.14	39.80	58.67	pass ( $p=0.0078$ )

TABLE II : PPL & Commonsense Reasoning zero-shot performance of the pruned LLaMA-7B. The PPL is evaluated on Wikitext2 and the Avg. denotes mean accuracy across BoolQ–OBQA.

Fig. 4 : Performance comparison of pruning methods on BERT-base under various FLOPs reduction ratios.

Sparsity	Method	QNLI	QQP	SST-2	CoLA	MRPC	RTE	Avg.
0%	BERT-base	91.40	91.50	93.23	58.90	88.48	66.80	81.71
60%	CoFi	87.86	90.56	89.95	47.76	82.85	55.91	75.81
	PGB	89.53	91.12	91.57	49.55	83.87	61.43	77.84
	RECAP	89.01	88.72	90.95	54.35	85.44	64.99	78.91
	REP-LIE(ours)	91.25	91.34	92.56	56.02	87.00	65.34	80.58
88%	CoFi	84.70	89.80	89.00	32.10	75.30	52.40	70.55

TABLE III : GLUE benchmark results at different high sparsity ratios.

Sparsity	Method	CoLA	SST-2	MRPC	QQP	QNLI	RTE	Memory (MB)
0%	BERT-base	58.90	93.20	88.50	90.89	91.40	66.80	1553
50%	Head-FT	51.10	89.75	85.44	88.05	90.66	64.99	524
	Pre-FT	53.17	91.16	83.88	89.47	90.12	62.45	957
	Post-FT	57.78	92.70	88.20	88.52	90.89	66.89	1553
	RECAP	56.56	92.20	86.56	89.39	90.74	65.98	702
	REP-LIE(ours)	57.23	92.33	87.90	90.20	90.96	66.45	476

TABLE IV: Comparison of model performance and GPU memory footprint(MB) for BERT-base between REP-LIE and other compression strategies.

Pruning criterion	QQP	QNLI	SST-2	CoLA	MRPC	RTE	MNLI	STSB	Avg.	Time (s)	Mem. (MB)	FLOPs (G)
Base model (dense)	91.50	91.40	93.20	58.90	88.50	66.80	84.60	88.90	82.97	–	1553	–
+ Weight magnitude	90.71	89.10	92.00	54.40	86.20	65.63	83.12	84.60	80.72	115.6	476	0.00
+ Gradient-based	91.12	90.71	92.50	57.43	86.96	66.50	83.85	86.50	81.87	180.8	810	0.26
+ LoRA-based (Ours)	91.33	90.96	92.33	57.23	87.90	66.45	83.98	87.01	82.14	120.4	476	0.01

TABLE V : Ablation study on pruning criterion for BERT-base.

Pruning criterion	PPL $\downarrow$	BoolQ	PIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Avg.	Time (s)	Mem. (MB)	FLOPs (G)
Base model (dense)	12.63	75.08	79.16	76.20	70.00	72.89	44.88	44.40	66.09	–	12884	–
+ Weight magnitude	32.59	60.21	68.88	58.25	57.62	56.56	29.73	38.20	52.79	982.3	5459	0.00
+ Gradient-based	30.45	69.88	72.53	59.83	60.23	58.89	31.62	40.98	56.28	1948.7	11601	13.40
+ LoRA-based (Ours)	28.46	70.58	73.26	60.23	60.98	58.52	32.40	40.02	56.57	983.6	5459	0.65

TABLE VI : Ablation study on pruning criterion for LLaMA-7B. The PPL is evaluated on Wikitext2 and the Avg. is computed across seven Commonsense Reasoning datasets.

Pruning criterion	PPL $\downarrow$	BoolQ	PIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Avg.	Time (s)	Mem. (MB)	FLOPs (G)
Base model (dense)	5.25	80.94	82.80	81.30	75.30	80.00	55.50	45.40	71.60	–	11263	–
+ Weight magnitude	26.64	63.45	70.76	61.26	60.87	68.52	35.26	35.29	56.49	986.8	5392	0.00
+ Gradient-based	18.12	74.23	77.53	74.86	68.01	73.13	41.62	38.98	64.05	2089.6	10589	14.48
+ LoRA-based (Ours)	15.36	75.11	77.60	74.01	69.27	73.37	40.83	39.20	64.19	996.9	5392	0.70

TABLE VII : Ablation study on pruning criterion for Mistral-7B. The PPL is evaluated on Wikitext2 and the Avg. is computed across seven Commonsense Reasoning datasets.

Method	Runtime @33%sparsity	Runtime @50%sparsity	Memory (MB)
LLM-Pruner	895.4s	1290.8s	6890
SlimGPT	783.0s	1074.0s	7375
REP-LIE (ours)	594.2s	983.6s	5459

TABLE VIII : Pruning runtime and peak GPU memory usage of REP-LIE and two advanced methods on LLaMA-7B.

Model	Runtime @33%sparsity	Runtime @50%sparsity	Memory (MB)
BERT-base	86.2s	120.4s	476
LLaMA-7B	594.2s	983.6s	5459
Mistral-7B	598.6s	996.9s	5392

TABLE IX : Pruning runtime and peak GPU memory usage of REP-LIE at different sparsity ratios on different models.

Sparsity	#Params	Latency (ms)	Throughput (token/s)	Memory (MB)	FLOPs (T)
0%	6.7B	14.87	8608	28796	1.7
33%	5.4B	12.57	10183	24697	1.4
50%	3.4B	8.87	14430	13587	0.9

TABLE X : Inference efficiency of pruned LLaMA-7B at different sparsity ratios.

Fig. 5 : Performance comparison of three pruning criteria on BERT-base under various FLOPs reduction ratios.

Fig. 6 : PPL & Commonsense Reasoning zero-shot performance comparison of three pruning criteria on LLaMA-7B under various FLOPs reduction ratios.

Fig. 7 : PPL & Commonsense Reasoning zero-shot performance comparison of three pruning criteria on Mistral-7B under various FLOPs reduction ratios.

Fig. 8 : The visualization of the average number of unpruned attention heads in MHA layers and unpruned channels in FFN layers under different sparsities.

Fig. 9 : The mean spearman correlation between low-rank and full weight gradients across different ranks and models.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 168 papers

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
Nov 8, 2023Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2LVLM Pruning LLM Pruning
Mohamed bin Zayed University of AI · Carnegie Mellon University
CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search
Jun 1, 2026Cheonjun ParkLLM Pruning LLM Compression
Hankuk University of Foreign Studies
Your Transformer Is Secretly Elastic
Oct 14, 2026Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts +1Elastic Neural Networks Transformer Pruning
University of Amsterdam · TNO, Intelligent Imaging

Nov 8, 2023J/K move · Enter open · S save

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma+1

Mohamed bin Zayed University of AI · Carnegie Mellon University

LVLM Pruning LLM Pruning Neural Network Pruning LLM Compression Transformer Pruning

Open paper AskPDF arXiv

Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.

Abstract

LLM Pruning

Explore similar work

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search

Your Transformer Is Secretly Elastic