GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

Xiaoyun Liu; Divya Saxena; Jiannong Cao; Yuqing Zhao; Yiying Dong; Penghui Ruan

cs.LGMar 12, 2026

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

Authors: Xiaoyun Liu, Divya Saxena, Jiannong Cao, Yuqing Zhao, Yiying Dong, Penghui Ruan

Organizations: Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India

Abstract

Structured pruning is widely applied to compress large language models (LLMs), but its performance depends heavily on how neuron importance is estimated. Most existing methods rely on activation statistics from a single calibration set, which introduces calibration bias and degrades downstream cross-task generalization. We observe that neurons exhibit heterogeneous distribution sensitivity, ranging from maintaining relatively stable rankings across calibration datasets to showing substantially larger cross-dataset variation. Ignoring this heterogeneity, existing methods rank all neurons in shared spaces with a uniform scoring source, so calibration-specific neurons dominate the ranking and weakly-activated neurons are scored unreliably. To address this, we propose GPrune-LLM, a structured pruning framework that reduces calibration bias by measuring and exploiting the cross-distribution behavior of neurons for fair comparison. Specifically, we restructure the neuron ranking space into behavior-consistent local spaces, adapt the scoring source where the calibration signal is unreliable, and learn per-module sparsity allocation under a global budget. Experiments across multiple models and downstream tasks show that GPrune-LLM improves the generalization of its base pruning metrics, with gains most pronounced at high sparsity, and reduces dependence on the choice of importance metric.

Figures & tables

Figure 1: Neuron importance changes with the calibration distribution, and neurons differ in their sensitivity to it. (a) Per-neuron importance rank drift

d

under a calibration swap, over all FFN layers. Re-sampling the same distribution (grey) barely shifts ranks (median rank correlation

0.98

), whereas switching WikiText2

\leftrightarrow

PTB (blue) causes large and heterogeneous drift (

0.62

), with some neurons drifting sharply while others remain nearly fixed. (b) Per-layer agreement (Adjusted Rand Index, ARI) between the neuron partitions produced before and after the same calibration swap. Modularizing neurons by cross-distribution behavior (GPrune) is more consistent than functional grouping estimated from a single distribution (FANG) (mean ARI

0.51

vs.

0.12

Figure 2: Overview of GPrune-LLM. Neurons are first partitioned into behavior-consistent modules by jointly modeling data-free parameter similarity and cross-distribution rank drift, with assignments refined differentiably. Each module is characterized by its mean rank drift (distribution sensitivity) and mean score (calibration-signal strength). Modules that are both distribution-sensitive and weakly supported are re-scored from the calibration distribution whose local ranking is more repeatable. Finally, module-specific sparsity is adaptively learned under a global budget with cross-distribution supervision, yielding a pruned LLM with improved generalization across diverse downstream tasks.

Ratio	Method	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	BoolQ	PIQA	Hella.	Wino.	ARC-e	ARC-c	OBQA	Avg. Acc. $\uparrow$
100%	LLaMA2-7B	12.19	48.35	71.10	78.40	72.96	67.25	69.36	40.53	40.80	62.91
80%	LLM-Pruner	19.23	72.62	62.26	76.28	65.82	60.85	63.85	37.46	39.20	57.96
	Wanda-sp	17.50	67.01	46.79	76.50	68.13	65.43	65.95	39.76	38.20	57.25
	GPrune-LLM (Wanda-sp)	15.44 $\uparrow$	56.95 $\uparrow$	60.76 $\uparrow$	76.22	68.67 $\uparrow$	65.51 $\uparrow$	64.18	37.80	39.80 $\uparrow$	58.99 $\uparrow$
	FLAP	16.41	61.61	52.45	73.61	62.56	62.12	59.13	34.81	39.20	54.84
	GPrune-LLM (FLAP)	15.55 $\uparrow$	59.56 $\uparrow$	64.16 $\uparrow$	74.76 $\uparrow$	66.49 $\uparrow$	62.19 $\uparrow$	62.50 $\uparrow$	36.35 $\uparrow$	39.40 $\uparrow$	57.98 $\uparrow$

Table 1: Comparison on LLaMA2-7B. For each retention ratio, the best and second-best results are marked in bold and underlined , respectively.

\uparrow

denotes improvement over the corresponding base method at the same retention ratio. For language modeling perplexity, lower is better. For classification tasks accuracy, higher is better.

Ratio	Method	WikiText2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg Acc $\uparrow$
100%	Vicuna1.5-7B	16.24	60.78	63.50
80%	LLM-Pruner	24.90	86.20	56.51
	Wanda-sp	23.66	73.67	59.08
	GPrune-LLM (Wanda-sp)	20.12 $\uparrow$	67.35 $\uparrow$	58.11
	FLAP	21.06	76.42	57.88
	GPrune-LLM (FLAP)	18.95 $\uparrow$	68.24 $\uparrow$	58.57 $\uparrow$

Table 2: Compact comparison on Vicuna1.5-7B.

Figure 3: Visualization of how GPrune-LLM changes FLAP’s pruning decisions on LLaMA2-7B at 50% retention. (a) GPrune-LLM shifts the retention budget away from the most unstable modules and toward the more stable ones. (b) GPrune-LLM concentrates its revision of which neurons to keep on the most unstable modules.

BCM	RASA	MSL	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg. Acc. $\uparrow$
			42.81	152.49	41.52
		✓	39.56	159.78	44.92
✓		✓	37.65	147.31	45.18
✓	✓	✓	37.02	125.41	46.40

Table 3: Component ablation of GPrune-LLM on FLAP at 50% retention ratio on LLaMA2-7B. BCM, RASA, and MSL denote Behavior-Consistent Modularization, Reliability-Aware Score Adaptation, and Modular Sparsity Learning, respectively.

Method	Calibration	WikiText2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg Acc $\uparrow$
FLAP	WikiText2	42.81	152.49	41.52
GPrune-LLM (FLAP)	WikiText2+ PTB	37.02 $\uparrow$	125.41 $\uparrow$	46.40 $\uparrow$
GPrune-LLM (FLAP)	WikiText2+ C4	40.93 $\uparrow$	136.11 $\uparrow$	46.61 $\uparrow$
FLAP	PTB	63.87	119.61	40.55
GPrune-LLM (FLAP)	PTB+ WikiText2	40.56 $\uparrow$	93.78 $\uparrow$	45.83 $\uparrow$
GPrune-LLM (FLAP)	PTB+ C4	55.04 $\uparrow$	117.85 $\uparrow$	46.07 $\uparrow$

Table 4: Comparison on different primary and auxiliary calibration datasets on FLAP at 50% retention ratio on LLaMA2-7B.

Method	Scoring	WikiText2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg Acc $\uparrow$
FANG	WT2	36.31	154.23	45.71
FANG	WT2+PTB mean score	43.01	115.92	42.18
FANG	WT2+PTB mean rank	50.29	144.71	42.77
FANG	WT2+PTB pool	42.88	114.20	42.04
GPrune-LLM (twFLAP)	WT2+PTB	36.83	129.77	47.22

Table 5: Effect of multi-dataset scoring at 50% retention ratio on LLaMA2-7B.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Ratio	Method	WikiText2 PPL $\downarrow$	PTB PPL $\downarrow$	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg. Acc. $\uparrow$
100%	Vicuna1.5-7B	16.24	60.78	75.75	77.80	71.02	67.64	69.02	40.87	42.40	63.50
80%	LLM-Pruner	24.90	86.20	50.98	75.84	64.14	59.83	66.12	38.65	40.00	56.51
	Wanda-sp	23.66	73.67	67.31	74.16	65.21	61.80	66.96	38.31	39.80	59.08
	GPrune-LLM (Wanda-sp)	20.12 $\uparrow$	67.35 $\uparrow$	56.76	75.95 $\uparrow$	66.85 $\uparrow$	65.11 $\uparrow$	65.95	36.95	39.20	58.11
	FLAP	21.06	76.42	66.88	72.47	63.17	64.48	63.55	34.81	39.80	57.88
	GPrune-LLM (FLAP)	18.95 $\uparrow$	68.24 $\uparrow$	59.11	75.03 $\uparrow$	67.26 $\uparrow$	65.27 $\uparrow$	65.57 $\uparrow$	36.18 $\uparrow$	41.60 $\uparrow$	58.57 $\uparrow$

Appendix

Table 6: Full comparison on Vicuna1.5-7B.

Ratio	Method	WikiText2 PPL $\downarrow$	PTB PPL $\downarrow$	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg. Acc. $\uparrow$
100%	LLaMA2-13B	10.98	54.42	68.99	79.05	76.60	69.77	73.32	45.56	42.00	65.04
50%	Wanda-sp	205.58	393.96	61.16	53.05	26.65	49.49	29.00	26.19	29.60	39.31
	GPrune-LLM (Wanda-sp)	28.30 $\uparrow$	126.67 $\uparrow$	62.17 $\uparrow$	68.17 $\uparrow$	54.20 $\uparrow$	55.09 $\uparrow$	51.35 $\uparrow$	33.62 $\uparrow$	39.00 $\uparrow$	51.94 $\uparrow$
	FLAP	27.91	156.91	62.66	65.02	48.95	57.85	47.43	31.48	37.00	50.06
	GPrune-LLM (FLAP)	26.54 $\uparrow$	105.20 $\uparrow$	61.59	67.52 $\uparrow$	53.71 $\uparrow$	56.75	52.15 $\uparrow$	32.85 $\uparrow$	38.20 $\uparrow$	51.82 $\uparrow$
	FANG	26.43	155.46	62.35	64.85	50.53	55.64	50.55	31.23	39.20	50.62

Appendix

Table 7: Full comparison on LLaMA2-13B. WikiText2 is abbreviated as WT2 elsewhere in the supplement. Throughout the supplement, bold and underlined values denote the best and second-best entries, respectively, and

\uparrow

indicates improvement over the corresponding base method. Lower perplexity and higher accuracy are better.

Component	Design dimension	Variant	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg. Acc. $\uparrow$
BCM	Modularization	Random	38.80	134.15	45.46
		Initialization	35.68	133.53	44.00
		Initialization + refinement	37.02	125.41	46.40
RASA	Module selection	None	37.65	147.31	45.18
		Random	36.59	136.71	45.00
		Rank drift	35.44	124.29	46.10

Appendix

Table 8: Fine-grained component ablations on LLaMA2-7B with FLAP at 50% retention ratio.

Method	Scoring	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg. Acc. $\uparrow$
Wanda-sp	WT2	262.32	505.51	39.37
Wanda-sp	Score avg.	111.15	309.30	40.37
Wanda-sp	Rank avg.	70.30	198.71	42.01
Wanda-sp	Pool	275.81	478.19	39.03
GPrune-LLM (Wanda-sp)	WT2+PTB	60.80	191.99	45.02
FLAP	WT2	42.81	152.49	41.52

Appendix

Table 9: Comparison of multi-dataset calibration at 50% retention ratio on LLaMA2-7B. Score avg. averages importance scores across WikiText2 and PTB. Rank avg. averages per-dataset ranks. Pool merges both calibration sets before scoring.

Dataset-pair comparison	ARI $\uparrow$	Matched overlap $\uparrow$
WT2–PTB vs. WT2–C4	$0.575\pm 0.121$	$0.707\pm 0.122$
WT2–PTB vs. PTB–C4	$0.582\pm 0.122$	$0.706\pm 0.124$
WT2–C4 vs. PTB–C4	$0.583\pm 0.124$	$0.714\pm 0.123$

Appendix

Table 10: Per-layer module agreement under calibration-pair substitution on LLaMA2-7B with FLAP. Values are mean

\pm

standard deviation across 32 FFN layers.

Retention	$\|\mathcal{D}\|$	Epoch	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg. Acc. $\uparrow$
50%	128	4	39.23	132.98	46.36
	256	7	39.56	145.52	45.86
	512	7	40.17	147.29	46.71
	1024	8	39.67	129.07	45.70
	2048	2	37.02	125.41	46.40

Appendix

Table 11: Effect of calibration sample number on GPrune-LLM (FLAP) at 50% retention ratio on LLaMA2-7B.

Method	Seed	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	Avg. Acc. $\uparrow$
FANG	0	36.31	154.23	45.71
	1	36.85	155.82	44.88
	2	37.55	158.75	45.50
	Mean $\pm$ Std.	$36.90\pm 0.51$	$156.27\pm 1.87$	$45.36\pm 0.35$
GPrune-LLM (twFLAP)	0	36.83	129.77	47.22
	1	36.16	146.90	45.65

Appendix

Table 12: Seed repeatability on LLaMA2-7B at 50% retention. Mean and standard deviation are computed over seeds 0–2.

Retention Ratio	Method	Params	Tokens/s $\uparrow$
100%	LLaMA2-7B	6.74B	97.04
50%	LLM-Pruner	3.45B	161.69
	Wanda-sp	3.5B	167.44
	GPrune-LLM (Wanda-sp)	3.49B	160.38
	FLAP	3.5B	148.52
	GPrune-LLM (FLAP)	3.49B	154.59

Appendix

Table 13: Practical inference throughput on LLaMA2-7B at 50% retention. All methods use the same device and software stack; reported values aggregate 10 timed repetitions after one warm-up run.

Component	Parameter	Value
General	Calibration samples for each of $A$ and $B$	2048
	Calibration sequence length	128
	Primary calibration dataset $A$	WikiText2
	Auxiliary calibration dataset $B$	PTB
BCM	Candidate initial module counts $\mathcal{K}$	16, 24, 32, 40, 48
RASA	Rank-drift quantile level $\gamma_{\mathrm{drift}}$	0.90

Appendix

Table 14: Principal hyperparameters used in the reported experiments.

Ratio	Method	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg. Acc. $\uparrow$
100%	LLaMA2-7B	12.19	48.35	71.10	76.82	72.96	67.25	53.58	40.53	40.80	60.43
80%	LLM-Pruner	19.23	72.62	62.26	75.41	65.82	60.85	50.17	37.46	39.20	55.88
	Wanda-sp	17.50	67.01	46.79	75.14	68.13	65.43	51.64	38.82	38.20	54.88
	GPrune-LLM (Wanda-sp)	15.44 $\uparrow$	56.95 $\uparrow$	60.76 $\uparrow$	74.54	68.67 $\uparrow$	65.51 $\uparrow$	50.55	37.80	39.80 $\uparrow$	56.80 $\uparrow$
	FLAP	16.41	61.61	52.45	73.01	62.56	62.12	47.35	34.81	39.20	53.07
	GPrune-LLM (FLAP)	15.55 $\uparrow$	59.56 $\uparrow$	64.16 $\uparrow$	74.76 $\uparrow$	66.49 $\uparrow$	62.19 $\uparrow$	49.37 $\uparrow$	36.35 $\uparrow$	39.40 $\uparrow$	56.10 $\uparrow$

Appendix

Table 15: Full comparison on LLaMA2-7B under the fixed task-metric reporting protocol.

Ratio	Method	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg. Acc. $\uparrow$
100%	Vicuna1.5-7B	16.24	60.78	75.75	77.80	71.02	67.64	56.06	39.93	42.40	61.52
80%	LLM-Pruner	24.90	86.20	50.98	75.24	64.14	59.83	53.07	37.12	40.00	54.34
	Wanda-sp	23.66	73.67	67.31	73.45	65.21	61.80	53.11	38.31	39.80	57.00
	GPrune-LLM (Wanda-sp)	20.12 $\uparrow$	67.35 $\uparrow$	56.76	74.59 $\uparrow$	66.85 $\uparrow$	65.11 $\uparrow$	53.07	36.35	39.20	55.99
	FLAP	21.06	76.42	66.88	71.71	63.17	64.48	51.47	34.73	39.80	56.04
	GPrune-LLM (FLAP)	18.95 $\uparrow$	68.24 $\uparrow$	59.11	74.54 $\uparrow$	67.26 $\uparrow$	65.27 $\uparrow$	52.57 $\uparrow$	35.41 $\uparrow$	41.60 $\uparrow$	56.54 $\uparrow$

Appendix

Table 16: Full comparison on Vicuna1.5-7B under the fixed task-metric reporting protocol.

Ratio	Method	WT2 PPL $\downarrow$	PTB PPL $\downarrow$	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg. Acc. $\uparrow$
100%	LLaMA2-13B	10.98	54.42	68.99	79.05	76.60	69.77	57.95	44.20	42.00	62.65
50%	Wanda-sp	205.58	393.96	61.16	50.87	26.65	49.49	28.66	26.19	29.60	38.95
	GPrune-LLM (Wanda-sp)	28.30 $\uparrow$	126.67 $\uparrow$	62.17 $\uparrow$	67.90 $\uparrow$	54.20 $\uparrow$	55.09 $\uparrow$	43.39 $\uparrow$	33.62 $\uparrow$	39.00 $\uparrow$	50.77 $\uparrow$
	FLAP	27.91	156.91	62.66	65.02	48.95	57.85	43.22	31.48	37.00	49.46
	GPrune-LLM (FLAP)	26.54 $\uparrow$	105.20 $\uparrow$	61.59	67.52 $\uparrow$	53.71 $\uparrow$	56.75	44.32 $\uparrow$	32.85 $\uparrow$	38.20 $\uparrow$	50.71 $\uparrow$
	FANG	26.43	155.46	62.35	64.85	50.53	55.64	43.60	31.23	39.20	49.63

Appendix

Table 17: Full comparison on LLaMA2-13B under the fixed task-metric reporting protocol.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Sep 10, 2026

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (Latent-Informed Layer Analysis) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~~pp in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22

\times

reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Sankar Behera, Dhruv Singh, Anshika Agnihotri +3

CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1

LLM Pruning LLM Compression Structured Pruning Neural Network Pruning

Ask PDF

Mar 25, 2026

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. At 20% sparsity on Gemma-2 2B, DIET improves average accuracy by 2.1 percentage points over SliceGPT and by 9.2 percentage points over PuDDing. On Gemma-2 9B, DIET remains competitive with SliceGPT while consistently outperforming PuDDing at 10% and 20% sparsity.

Jimyung Hong, Jaehyung Kim

Yonsei University

LLM Pruning Structured Pruning Neural Network Pruning LLM Compression

Ask PDF

Aug 23, 2026

Correlation-Aware Structured Pruning for Large Language Models

Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Many existing methods assess the importance of prunable units (e.g., channels or heads) in isolation, implicitly assuming that pruning errors are additive. This independence assumption is often invalidated by the non-orthogonality of model weights and strong correlations between unit activations, potentially leading to performance degradation. To address this, we propose a Correlation-Aware Structured Pruning method. We formulate the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.

Sicheng Xu, Hao Shi, Wei Zhang +6

Department of Mathematical Sciences, Tsinghua University · Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co. Ltd.

LLM Pruning Neural Network Pruning Structured Sparsity

Ask PDF

Abstract

LLM Pruning

Explore similar work

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Correlation-Aware Structured Pruning for Large Language Models