Correlation-Aware Structured Pruning for Large Language Models

Sicheng Xu; Hao Shi; Wei Zhang; Haoran Pang; Zhenyu Ming; Hao Wu; Zhongyi Huang; Xin Yao

cs.CLAug 23, 2026

Correlation-Aware Structured Pruning for Large Language Models

Authors: Sicheng Xu, Hao Shi, Wei Zhang, Haoran Pang, Zhenyu Ming, Hao Wu, Zhongyi Huang, Xin Yao,

Organizations: Department of Mathematical Sciences, Tsinghua University · Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co. Ltd.

Abstract

Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Many existing methods assess the importance of prunable units (e.g., channels or heads) in isolation, implicitly assuming that pruning errors are additive. This independence assumption is often invalidated by the non-orthogonality of model weights and strong correlations between unit activations, potentially leading to performance degradation. To address this, we propose a Correlation-Aware Structured Pruning method. We formulate the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.

Figures & tables

Figure 1: The Correlation-Aware Structured Pruning (CASP) workflow. Linear pipeline (left-right): (1) Layer-Wise Allocation generates a sparsity profile; (2) Correlation-Aware Modeling builds interaction matrix

Q

; (3) Greedy Algorithm selects optimal units to yield the pruned LLM.

Ratio	Method	WikiText2 $\downarrow$	PTB $\downarrow$	BoolQ $\uparrow$	PIQA $\uparrow$	HellaSwag $\uparrow$	WinoGrande $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Average $\uparrow$
0%	Ground Truth	6.9413	23.0732	0.8058	0.7617	0.5800	0.6835	0.7264	0.4172	0.3200	0.6135
10%	LLM-Pruner	8.3402	31.7846	0.7037	0.7693	0.5526	0.6780	0.7306	0.4249	0.3000	0.5942
	FLAP	19.0166	64.8369	0.6630	0.7595	0.5594	0.6725	0.6570	0.3788	0.2300	0.5600
	LoRAP	7.5938	27.6150	0.7621	0.7655	0.5685	0.6819	0.7155	0.4070	0.3120	0.6018
	CASP	7.3818	25.5896	0.7820	0.7699	0.5671	0.6827	0.7247	0.4130	0.3200	0.6085
20%	LLM-Pruner	12.5446	51.5916	0.5633	0.7476	0.5194	0.6606	0.7033	0.4087	0.3060	0.5584

Table 1: Zero-shot performance of the compressed LLaMA2-7B. The average score is computed across seven datasets. The " bolded " represents the best result under the same pruning ratio.

Ratio	Method	WikiText2 $\downarrow$	PTB $\downarrow$	BoolQ $\uparrow$	PIQA $\uparrow$	HellaSwag $\uparrow$	WinoGrande $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Average $\uparrow$
0%	Ground Truth	4.8838	28.9402	0.8214	0.7943	0.6019	0.7238	0.7896	0.4735	0.3460	0.6501
10%	LLM-Pruner	5.6887	34.9768	0.7333	0.7845	0.5838	0.7088	0.7748	0.4599	0.3460	0.6273
	FLAP	12.1349	59.7305	0.7673	0.7802	0.5688	0.7072	0.7386	0.4215	0.3420	0.6179
	LoRAP	5.9488	32.8577	0.8208	0.7878	0.6000	0.7206	0.7786	0.4590	0.3460	0.6447
	CASP	5.1634	32.7983	0.8196	0.7927	0.5971	0.7293	0.7824	0.4710	0.3420	0.6477
20%	LLM-Pruner	6.6494	45.8865	0.6514	0.7884	0.5619	0.7009	0.7580	0.4428	0.3320	0.6051

Table 2: Zero-shot performance of the compressed LLaMA2-13B. The average score is computed across seven datasets. The " bolded " represents the best result under the same pruning ratio.

Figure 2: Off-diagonal energy ratio across layers in LLaMA2-7B. We report the proportion of interaction (off-diagonal) terms in the reconstruction energy for both attention-head pruning and FFN-channel pruning. The consistently high ratio demonstrates that cross-unit interactions dominate the pruning error in most layers, invalidating the independence assumption.

Ratio	Method	WikiText2 $\downarrow$	PTB $\downarrow$	Avg.
30%	w/o Greedy	10.40	31.11	0.5211
	w/o Allocation	11.45	32.47	0.4114
	CASP	9.73	33.18	0.5493
50%	w/o Greedy	22.94	72.63	0.4088
	w/o Allocation	58.81	159.88	0.3399
	CASP	18.18	67.95	0.4492

Table 3: Ablation study of different components in our proposed method (CASP) at 30% and 50% pruning ratios. The " bolded " represents the best result under the same pruning ratio.

Figure 3: Comparison of Optimization Methods. Reconstruction error ( $\mathcal{L}$ ) on the first FFN layer of LLaMA2-7B under varying pruning ratios ( $r$ ). We plot the Diagonal baseline (blue), the theoretical Relaxation Lower Bound (black dotted), the projected Relaxation solution (green solid), and our Greedy interaction algorithm (red dashed).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Layer-wise gradient sensitivity in LLaMA2-7B. We compute each layer’s sensitivity as the squared $\ell_{2}$ norm of the gradient of the final objective with respect to the layer output, and report it across depth (linear and log scales). The sensitivity generally decreases with depth, motivating an increasing pruning ratio for deeper layers.

Ratio	Time (s)
$10\%$	184.45
$20\%$	185.33
$30\%$	185.35
$50\%$	186.59

Appendix

Table 4: Average pruning time of our method on LLaMA2-7B under different pruning ratios. Each result is averaged over

10

runs.

Ratio	Method	Params (B)	MACs(G)	Prefilling(s)	Decoding(s)
0%	Dense	6.74	845.71	0.321	2.694
20%	CASP	5.44	624.80	0.260	2.243
50%	CASP	3.51	397.62	0.175	1.870

Appendix

Table 5: Inference latency and parameter counts for LLaMA2-7B at different sparsity levels.

Ratio	Method	WikiText2 $\downarrow$	PTB $\downarrow$	BoolQ $\uparrow$	PIQA $\uparrow$	HellaSwag $\uparrow$	WinoGrande $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Average $\uparrow$
50% w/o tune	LoRAP	22.9549	129.2947	0.6058	0.6235	0.3591	0.5556	0.4141	0.2509	0.1960	0.4293
50% w/o tune	CASP	18.1812	67.9484	0.6034	0.6572	0.3762	0.5541	0.4596	0.2577	0.2360	0.4492
50% w/ tune	LoRAP	12.9873	59.5412	0.6181	0.6975	0.4226	0.5951	0.5610	0.2841	0.2460	0.4892
50% w/ tune	CASP	12.5192	42.4517	0.6386	0.6757	0.4349	0.5635	0.6232	0.2711	0.2480	0.4936

Appendix

Table 6: Performance recovery via fine-tuning on LLaMA2-7B at 50% sparsity. The average score is computed across seven datasets. The " bolded " represents the best result under the same pruning ratio.

Value	WikiText2	PTB	Avg.
$\alpha=0.5$	17.45	76.70	0.4074
$\alpha=1.0$	18.18	67.95	0.4492
$\alpha=1.5$	24.51	80.38	0.4255
$\alpha=2.0$	25.49	78.82	0.4384

Appendix

Table 7: Ablation study of temperature parameter

\alpha

in our proposed method (CASP) at 50% pruning ratios. The " bolded " represents the best result under the same pruning ratio.

Ratio	Method	WikiText2 $\downarrow$	PTB $\downarrow$	BoolQ $\uparrow$	PIQA $\uparrow$	HellaSwag $\uparrow$	WinoGrande $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Average $\uparrow$
0%	Ground Truth	6.7805	26.7653	0.8162	0.7688	0.5657	0.7080	0.7597	0.4428	0.3500	0.6302
10%	LLM-Pruner	8.3728	30.8669	0.7095	0.7535	0.5444	0.6772	0.7487	0.4300	0.3400	0.6005
	FLAP	18.1103	67.1568	0.7306	0.7524	0.5457	0.6827	0.7214	0.3993	0.3460	0.5969
	LoRAP	7.3890	29.7425	0.7991	0.7633	0.5609	0.6788	0.7472	0.4352	0.3380	0.6175
	CASP	7.2673	28.3805	0.7936	0.7715	0.5589	0.6819	0.7500	0.4394	0.3260	0.6173
20%	LLM-Pruner	11.4220	38.4145	0.6453	0.7503	0.5136	0.6314	0.7218	0.3899	0.3020	0.5649

Appendix

Table 8: Zero-shot performance of the compressed Vicuna-7B. The average score is computed across seven datasets. The " bolded " represents the best result under the same pruning ratio.

Ratio	Method	WikiText2 $\downarrow$	PTB $\downarrow$	BoolQ $\uparrow$	PIQA $\uparrow$	HellaSwag $\uparrow$	WinoGrande $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	OBQA $\uparrow$	Average $\uparrow$
0%	Ground Truth	6.1367	10.5469	0.8226	0.8063	0.7922	0.7364	0.7752	0.5410	0.4480	0.7031
10%	LLM-Pruner	8.7911	13.9107	0.6881	0.7671	0.7231	0.7151	0.6881	0.4437	0.4200	0.6350
	LoRAP	7.5508	12.4922	0.7783	0.7780	0.7503	0.7017	0.7369	0.5009	0.4240	0.6672
	CASP	7.1484	12.0156	0.8040	0.7878	0.7595	0.7261	0.7353	0.5000	0.4140	0.6752
50%	LLM-Pruner	60.0247	94.5353	0.5180	0.5696	0.3351	0.5130	0.3729	0.2432	0.2900	0.4060
	LoRAP	92.8750	188.3750	0.5138	0.6507	0.4229	0.5399	0.3906	0.2594	0.2980	0.4393

Appendix

Table 9: Zero-shot performance of the compressed LLaMA3-8B. The average score is computed across seven datasets. The " bolded " represents the best result under the same pruning ratio.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Mar 25, 2026Jimyung Hong, Jaehyung KimLLM Pruning Structured Pruning
Yonsei University
GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
Mar 12, 2026Xiaoyun Liu, Divya Saxena, Jiannong Cao +3LLM Pruning Structured Pruning
Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India
SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning
Oct 11, 2026Sankar Behera, Dhruv Singh, Satyadev Ahlawat +1LLM Pruning Structured Pruning
Department of Computer Science and Engineering, IIT Jammu · Department of Mathematics, IIT Jammu · Department of Electrical Engineering, IIT Jammu

Mar 25, 2026J/K move · Enter open · S save

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Jimyung Hong, Jaehyung Kim

Yonsei University

LLM Pruning Structured Pruning Neural Network Pruning LLM Compression

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. At 20% sparsity on Gemma-2 2B, DIET improves average accuracy by 2.1 percentage points over SliceGPT and by 9.2 percentage points over PuDDing. On Gemma-2 9B, DIET remains competitive with SliceGPT while consistently outperforming PuDDing at 10% and 20% sparsity.

Abstract

LLM Pruning

Explore similar work

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning