Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Rocktim Jyoti Das; Mukul Ranjan; Mingjie Sun; Liqun Ma; Zhiqiang Shen

cs.CLNov 8, 2023

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Authors: Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma, Zhiqiang Shen

Organizations: Mohamed bin Zayed University of AI · Carnegie Mellon University

Abstract

Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.

Figures & tables

Symbol	Dimension	Description
$\mathbf{W}$	$\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$	Weight matrix of a neural network layer
$\mathbf{X}$	$\mathbb{R}^{N\times L\times d_{\text{in}}}$	Input activation matrix across samples
$\mathbf{G}$	$\mathbb{R}^{N\times d_{\text{out}}\times d_{\text{in}}}$	Gradient matrix, $\mathbf{G}[k,i,j]=\frac{\partial E}{\partial\mathbf{W}[i,j]}\big\|_{k}$
$\mathbf{S}$	$\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$	Importance score matrix (pruning metric)
$\mathbf{M}$	$\{0,1\}^{d_{\text{out}}\times d_{\text{in}}}$	Binary pruning mask
$\mathbf{H}$	$\mathbb{R}^{d\times d}$	Hessian matrix, $\mathbf{H}=\frac{\partial^{2}E}{\partial\mathbf{w}^{2}}$

Table 1

Figure 1: Illustration of our method GBLM-Pruner . Given a weight matrix, $\mathbf{W}$ , a gradient matrix, $\mathbf{G}$ , and an input feature activation, $\mathbf{X}$ , weight importance is computed as an element-wise multiplication of weight magnitude and $\ell_{1}$ or $\ell_{2}$ norm of the gradients across multiple samples, denoted as $\|\mathbf{G}\|_{p}\cdot|\mathbf{W}|$ , optionally, it is promotable to add the multiplication of weight and the $\ell_{2}$ norm of input activations, denoted as $|\mathbf{W}|\cdot\|\mathbf{X}\|_{2}$ .

Models	Method	BoolQ	RTE	HellaSwag	WinoGrande	ARC-e	OBQA	Mean
LLaMA-1-7B	Dense	75.11	66.43	76.21	69.85	72.81	44.40	67.47
	Magnitude	54.65	54.15	60.90	59.43	54.38	35.20	53.12
	SparseGPT	72.87	53.07	69.77	67.88	66.46	40.60	61.77
	Wanda	71.25	54.87	70.12	66.06	65.11	39.60	61.17
	Ours	73.43	59.93	70.29	67.40	65.99	41.40	63.07
LLaMA-1-13B	Dense	77.98	70.40	79.07	72.77	74.75	44.80	69.96

Table 1: Zero-Shot evaluation on 50

\%

unstructured sparsity pruned models for different benchmarks.

Comparison Group	Perplexity
layer	7.45
input,1	10.16
input,128	7.64
output,1	6.86
output,128	7.47

Table 2: Comparison group for GBLM-Pruner .

		LLaMA-2			LLaMA-1
Method	Sparsity	7B	13B	70B	7B	13B	30B
None	0	5.47	4.88	3.32	5.68	5.09	4.10
Magnitude	0.5	16.03	6.83	5.36	17.29	20.21	7.54
SparseGPT	0.5	7.00	6.03	4.25	7.22	6.19	5.32
Wanda	0.5	6.92	5.97	4.22	7.26	6.15	5.24
GBLM-Pruner ℓ1	0.5	6.86	5.88	4.17	7.15	6.11	5.18

Table 3: WikiText perplexity of pruned LLaMA-1 and LLaMA-2 family of models.

Model	Methods	MMLU	GPQA-D	GPQA	MATH	GSM8K	Mean
		(5-s)	(5-s)	(5-s)	(4-s)	(5-s)
Llama-3.1-8B	Base	65.20	24.24	27.68	8.54	51.55	35.44
	SparseGPT	53.21	23.23	22.32	7.76	16.60	24.63
	Wanda	49.34	24.18	23.66	7.92	13.57	23.73
	Ours	50.16	25.25	25.00	8.54	14.86	24.76
Qwen3 8B	Base	74.95	19.70	28.57	25.70	87.34	47.25

Table 4: Performance comparison of base and pruned language models with 50% unstructured sparsity on STEM and general purpose reasoning benchmarks.

Model	Method	MBPP	HumanEval	Mean
		(3-shot)	(0-shot)
Qwen3-8B	Base	65.60	62.80	64.20
	SparseGPT	37.20	12.80	25.00
	Wanda	43.40	26.22	34.81
	Ours	44.00	28.05	36.02
Qwen3-14B	Base	71.60	55.49	63.54

Table 5: Performance comparison of base and pruned language models on coding benchmarks. All pruned models use 50% unstructured sparsity.

Model	Method	MMMU	MTVQA	Mean
Qwen2.5-VL-3B	Base	48.60	24.34	36.47
	SparseGPT	30.33	19.66	25.00
	Wanda	37.50	19.96	28.73
	Ours	36.81	20.97	28.89
Qwen2.5-VL-7B	Base	49.55	29.59	39.57
	SparseGPT	39.33	27.14	33.24

Table 6: Performance comparison of base and pruned vision-language models on multimodal benchmarks at 50% unstructured sparsity. Values are shown as percentages. Bold indicates best performance among pruning methods, underlined indicates second best.

Method	Sparsity	7B	13B
Magnitude	0.5	16.03	6.83
SparseGPT	0.5	7.00	6.03
Wanda	0.5	6.92	5.97
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{1}$ (Ours)	0.5	7.17	6.15
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{2}$ (Ours)	0.5	7.09	5.96

Table 7: Ablation on the pruning metrics.

Method	Sparsity	Perplexity
$\left\|\textbf{W}\right\|\cdot\left\|\textbf{G}_{acc}\right\|$	0.5	119.72
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	7.17
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{2}$	0.5	7.09
$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ + $\alpha\cdot\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	6.90
$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ + $\alpha\cdot\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{2}$	0.5	6.88
$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ - $\alpha\cdot\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	9743.65

Table 8: Pruning metric on weight, gradient, activation.

Panel of Figure 2 — Figure 2: Illustration of learned pruning pattern.

Model	Memory (GB)	Time (seconds)
LLaMA-7B	62	71.3
LLaMA-13B	96	102.1

Table 9: Computation requirements on an A100 GPU.

Method	LLaMA-7B	LLaMA-13B
SparseGPT	256,732.95	423,550.22
Wanda	257.04	455.39
GBLM-Pruner	403.76	561.30

Table 10: Pruning metric computation and weight removal time (milliseconds) on an A100 GPU.

LLaMA-1 layer	Dense	2:4	Speedup
q/k/v/o_proj	3.49	2.14	1.63 $\times$
up/gate_proj	9.82	6.10	1.61 $\times$
down_proj	9.92	6.45	1.54 $\times$

Table 11: Inference speedup for 2:4 sparsity on an A6000 GPU across transformer layers of LLaMA-1.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Sparsity	Wanda	Magnitude	Ours $\ell_{1}$	Ours $\ell_{2}$
0	75.40	75.40	75.40	75.40
0.5	64.54	59.48	64.64	64.86
0.6	43.65	29.98	44.15	44.23
0.7	7.92	1.88	8.89	8.02
0.8	0.20	0.18	0.32	0.24

Appendix

Table 12: ViT-B model pruning.

Scaling Factor, ( $\alpha$ )	Perplexity
0.001	6.920
0.01	6.919
0.1	6.921
1	6.912
10	6.890
100	6.858

Appendix

Table 13: Ablation of scaling factor.

Figure 3: Robustness to calibration samples.

Panel of Figure 4 — Figure 4: Sparsity variation results for a large and a small model where we compare the performance of our method against other baseline methods.

Method	Sparsity	LLaMA-2-7B-chat	LLaMA-13B-chat
None	0	7.08	6.11
Magnitude	0.5	22.82	8.49
SparseGPT	0.5	8.66	7.26
Wanda	0.5	8.78	7.50
GBLM-Pruner ℓ2	0.5	8.52	7.27
GBLM-Pruner ℓ1	0.5	8.40	7.10

Appendix

Table 14: WikiText validation perplexity of different pruning methods for LLaMA 2 chat models.

Method	Datasplit	no	yes
		Weight Update
Magnitude	Calib	18.14	12.93
Magnitude	Valid	17.29	12.55
Wanda	Calib	7.52	7.61
Wanda	Valid	7.26	7.36
Ours	Calib	7.54	7.64

Appendix

Table 15: OBS weight update.

Method	Sparsity	Perplexity	Method	Sparsity	Perplexity
$\left\|\textbf{W}\right\|\cdot\left\|\textbf{G}_{acc}\right\|$	0.5	119.72	$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ + $\alpha\cdot\textbf{W}\cdot\textbf{G}_{acc}$	0.5	7.04
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	7.17	$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ + $\alpha\cdot\textbf{W}\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	180490.19
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{G}\right\\|_{2}$	0.5	7.09	$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ + $\alpha\cdot\textbf{W}\cdot\left\\|\textbf{G}\right\\|_{2}$	0.5	91781.49
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2}\cdot\left\|\textbf{G}_{acc}\right\|$	0.5	69.59	$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ - $\alpha\cdot\textbf{W}\cdot\textbf{G}_{acc}$	0.5	7.14
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2}\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	7.31	$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ - $\alpha\cdot\textbf{W}\cdot\left\\|\textbf{G}\right\\|_{1}$	0.5	246846.28
$\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2}\cdot\left\\|\textbf{G}\right\\|_{2}$	0.5	7.31	$(\left\|\textbf{W}\right\|\cdot\left\\|\textbf{X}\right\\|_{2})^{2}$ - $\alpha\cdot\textbf{W}\cdot\left\\|\textbf{G}\right\\|_{2}$	0.5	283620.75

Appendix

Table 16: Pruning metric.

Model	Sparsity	Wanda	GBLM-Pruner
Qwen3-14B	0.7	33.12	32.13
	0.8	254.90	192.85
	0.9	10902.73	5225.87
	0.95	16872.77	11845.89
LLaMA-3.1-8B	0.7	113.59	111.48
	0.8	1480.43	835.78

Appendix

Table 17: WikiText perplexity at higher sparsity levels. Bold indicates best performance.

LVLM Pruning

See all 35 papers

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models
Apr 21, 2026Ziyang Wang, Jiangfeng Xiao, Chuan Xiao +3LLM Pruning LLM Compression
Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Mar 25, 2026Jimyung Hong, Jaehyung KimLLM Pruning Structured Pruning
Yonsei University
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1

Apr 21, 2026J/K move · Enter open · S save

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li+2

Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University

LLM Pruning LLM Compression Structured Pruning

Open paper AskPDF arXiv

Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a single global budget. Instead of learning importance scores without constraints and applying the budget only after training, GRASPrune learns lightweight gate scores with a projected straight-through estimator that enforces a hard mask satisfying the budget at every step while keeping the backbone weights frozen. After the mask is fixed, we calibrate scaling factors on the retained units to mitigate scale mismatch caused by pruning, and fold these factors into the pruned weights to obtain a smaller dense checkpoint with no extra parameters at inference. On LLaMA-2-7B, GRASPrune removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five benchmarks, using four epochs on 512 unlabeled calibration sequences on a single NVIDIA A100 80GB GPU without any full model fine-tuning.

Abstract

LVLM Pruning

Explore similar work

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Prune, Update and Trim: Robust Structured Pruning for Large Language Models