GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation

Hao Liu; Guangyan Li; Wensheng Zhang; Yongqiang Tang

cs.CLFeb 16, 2026

GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation

Authors: Hao Liu, Guangyan Li, Wensheng Zhang, Yongqiang Tang

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Guangzhou University, Guangzhou, China

Abstract

Large Language Models (LLMs) exhibit strong reasoning abilities, but their high computational costs limit their practical deployment. Recent studies reveal significant redundancy in LLMs layers, making layer pruning an active research topic. Layer pruning research primarily focuses on two aspects: measuring layer importance and recovering performance after pruning. Unfortunately, the present works fail to simultaneously maintain pruning performance and efficiency. In this study, we propose GradMAP, a faster layer pruning method with Gradient Metric And Projection compensation, which consists of two stages. In the first stage, we introduce a novel metric based on gradient magnitudes, enabling a global assessment of layer importance. Note that, it requires only a single backward propagation step per pruning decision, substantially enhancing pruning efficiency. In the second stage, we first analyze the layers with the largest mean shift resulting from pruning, and then incorporate a simple yet effective projection compensation matrix to correct this drift in one step. In this way, the degradation of model performance caused by layer pruning is effectively alleviated. Extensive experiments show that GradMAP outperforms previous layer pruning methods in both pruning speed (achieving an average $4\times$ speedup) and performance.

Figures & tables

Figure 1: Comparison of pruning time and accuracy across different methods on the Vicuna-7B.

Figure 2: The Stage 1 of GradMAP introduces a novel gradient-based metric for estimating layer importance in LLMs, which quantifies each layer’s contribution by analyzing gradient magnitudes. This metric enables the iterative identification and pruning of unimportant layers.

Figure 3: The Stage 2 of GradMAP. To mitigate capacity loss, we introduce a learnable projection compensation matrix that optimizes reconstruction on a small calibration dataset.

Ratio	Method	Average Accuracy $\uparrow$			PPL $\downarrow$			Pruning Time $\downarrow$
Ratio	Method	LLaMA2-7B	LLaMA2-13B	Vicuna-7B	LLaMA2-7B	LLaMA2-13B	Vicuna-7B	LLaMA2-7B	LLaMA2-13B	Vicuna-7B
0%	Dense	66.79	69.31	66.85	12.18	10.98	16.23	–	–	–
12.50%	LLM-Pruner	58.96	63.55	60.64	16.27	14.67	20.47	–	–	–
	Subcloning	59.92	61.25	61.48	24.26	19.91	27.55	–	–	–
	SLEB	60.87	60.60	62.79	16.68	14.63	21.68	503.98	1638.51	506.11
	ShortGPT	62.32	64.14	63.21	16.85	14.78	21.63	519.33	1682.42	523.34

Table 1: Zero-shot evaluations of different pruning methods with 12.5%, 25%, and 40.63% pruning ratios across various LLMs. ‘PPL’ means perplexity on Wikitext2. ‘bold’ indicates the best performance. † denotes models pruned using only Stage 1 (layer selection), while ‡ indicates models further refined by Stage 2 (projection compensation). Pruning time in seconds.

Type	Method	Ratio	Throughput (Tokens/s)	Latency (ms)
–	Dense	0%	299 (1.00 $\times$ )	1718.4 (1.00 $\times$ )
Unstructured	Wanda	50%	293 (0.98 $\times$ )	1555.5 (1.10 $\times$ )
Unstructured	SparseGPT	50%	293 (0.98 $\times$ )	1555.5 (1.10 $\times$ )
Structured	LLM-Pruner	20%	314 (1.05 $\times$ )	1534.3 (1.12 $\times$ )
	SliceGPT	20%	314 (1.05 $\times$ )	1658.7 (1.04 $\times$ )
	SliceGPT	25%	331 (1.11 $\times$ )	1440.7 (1.19 $\times$ )

Table 2: Throughput and latency comparison under different pruning types on LLaMA2-7B.

Panel of Figure 4 — Figure 4: Perplexity of LLaMA2-7B under varying compression ratios.

Model	Method	Ratio	Benchmarks												Average
Model	Method	Ratio	C3	CMNLI	CHID	BoolQ	WSC	HeSW	PIQA	CoQA	Race-M	Race-H	MMLU	CMMLU	Average
LLaMA3.1-8B	Dense	0.00%	54.08	32.98	36.99	69.79	71.10	74.66	80.96	71.42	70.61	63.15	66.74	50.87	59.07
	SLEB	25.0%	37.15	33.10	11.79	54.43	62.50	48.96	70.46	39.72	21.31	21.38	25.29	25.29	37.62
	ShortGPT	25.0%	43.23	33.13	7.09	57.86	63.50	55.71	69.21	54.22	35.17	35.17	56.69	35.12	45.50
	GradMAP †	25.0%	47.07	32.72	26.07	53.15	63.50	58.54	73.45	59.13	40.60	35.48	54.98	39.60	48.68
	GradMAP ‡	25.0%	47.67	33.43	26.97	53.79	62.50	60.05	72.91	58.72	40.74	33.85	55.63	40.77	48.92

Table 3: Accuracy of different pruning methods across classification benchmarks on LLaMA3.1-8B, Baichuan2-7B, and OPT-6.7B.

Figure 5: GPU memory consumption and running time of GradMAP during Stage 1 (importance estimation) and Stage 2 (projection compensation) under a 25% compression ratio.

Model	Method	BoolQ $\uparrow$	PIQA $\uparrow$	HellaS. $\uparrow$	WinoG. $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Avg. $\uparrow$	$\Delta$
LLaMA2-7B	SLEB	61.38	72.25	62.27	60.85	63.34	33.53	38.00	55.96	+0.79
	w/ Ours Stage2	67.55	71.98	62.30	60.69	63.05	33.87	37.80	56.75	+0.79
	ShortGPT	53.64	68.17	62.50	65.98	55.77	34.64	38.60	54.19	+0.84
	w/ Ours Stage2	58.62	68.44	62.15	65.59	56.78	34.81	38.80	55.03	+0.84
	GradMAP ‡	70.46	70.95	63.91	65.35	64.02	37.46	38.00	58.59	–
Vicuna-7B	SLEB	66.64	70.29	60.17	55.96	62.54	33.79	34.80	54.88	+0.09

Table 4: Performance of integrating our projection compensation matrix with existing layer pruning methods on LLaMA2-7B and Vicuna-7B under 25% compression ratio.

\Delta

indicates improvement after applying our Stage 2.

Figure 6: Comparison of layer selection strategies for GradMAP and other baselines across different compression ratios.

Method	MMLU $\uparrow$	CMMLU $\uparrow$	GSM8k $\uparrow$	XSum $\uparrow$	StrategyQA $\uparrow$	Avg. $\uparrow$
SLEB	25.42	22.36	0.89	0.15	18.47	13.46
ShortGPT	28.59	40.62	7.89	1.67	20.13	19.78
GradMAP †	30.77	48.89	11.45	1.67	40.74	26.70
GradMAP ‡	30.17	51.84	11.51	2.12	40.79	27.20

Table 5: Performance comparison on additional datasets and generative tasks using Baichuan2-7B.

Figure 7: Distribution of first-order moment offsets before and after pruning for OPT-6.7B and OPT-2.7B.

Figure 8: Distribution of first-order moment offsets before and after pruning for LLaMA2-7B and Vicuna-7B.

Panel of Figure 9 — Figure 9: Accuracy comparison on different MMLU dataset subjects during pruning. Subjects: A: High School European History, B: High School Government and Politics, C: Medical Genetics, D: Management, E: High School Geography.

Figure 10: Performance comparison of GradMAP using different calibration datasets across multiple benchmarks.

Component	Variant	LLaMA2-7B	Vicuna-7B
Layer Selection	One-Shot	50.07	50.92
Layer Selection	GradMAP †	56.08	55.88
Compensation	Local	56.66	55.57
	Top-5	57.95	56.31
	Top-2	58.21	57.70
	GradMAP ‡	58.59	58.15

Table 6: Ablation study of layer selection and compensation strategies.

Types	PPL $\downarrow$	BoolQ $\uparrow$	PIQA $\uparrow$	HellaS. $\uparrow$	WinoG. $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Avg. $\uparrow$
MHA Weights
$W_{Q}$	28.49	50.98	70.73	61.17	61.80	63.01	37.80	38.20	54.81
$W_{K}$	28.49	50.98	70.73	61.17	61.80	63.01	37.80	38.20	54.81
$W_{V}$	28.54	51.65	70.51	61.10	61.80	63.09	38.31	38.20	54.95
$W_{O}$	27.94	56.36	70.67	61.00	61.72	63.09	37.88	37.40	55.45
FFN Weights

Table 7: Ablation of compensation on different weight matrices for Vicuna-7B at 25% compression ratio.

Model	$\bm{\mathcal{L}_{\mathrm{MSE}}}$	$\bm{\mathcal{L}_{\mathrm{reg}}}$	Avg. $\uparrow$
LLaMA2-7B	✓	✗	56.54
	✗	✓	56.66
	✓	✓	58.59
Vicuna-7B	✓	✗	56.98
	✗	✓	55.32
	✓	✓	58.15

Table 8: Ablation study of loss functions.

Model	Type	Time $\downarrow$	Accuracy $\uparrow$
LLaMA2-7B	Right	1475.1	58.70
LLaMA2-7B	Left	54.0	58.59
LLaMA2-13B	Right	2758.3	59.48
LLaMA2-13B	Left	93.1	59.49
Vicuna-7B	Right	1487.2	58.05
Vicuna-7B	Left	80.6	58.15

Table 9: Comparison of different calibration types on zero-shot performance.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
Nov 8, 2023Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2LVLM Pruning LLM Pruning
Mohamed bin Zayed University of AI · Carnegie Mellon University
SimDiff: Depth Pruning via Similarity and Difference
Apr 21, 2026Yuli Chen, Shuhao Zhang, Fanshen Meng +4LLM Pruning LLM Inference Acceleration
LOOP: Recovery-Aligned Iterative Layer Pruning via On-Policy Distillation
Oct 19, 2026Liang Xu, Shufan Shen, Zhongni Hou +4LLM Pruning Neural Network Pruning
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan

Nov 8, 2023J/K move · Enter open · S save

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma+1

Mohamed bin Zayed University of AI · Carnegie Mellon University

LVLM Pruning LLM Pruning Neural Network Pruning LLM Compression Transformer Pruning

Open paper AskPDF arXiv

Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.

Abstract

LLM Pruning

Explore similar work

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

SimDiff: Depth Pruning via Similarity and Difference

LOOP: Recovery-Aligned Iterative Layer Pruning via On-Policy Distillation