COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

Peiqi Yu; Nam Ling; Wei Wang; Wei Jiang

cs.LGAug 21, 2026

COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

Authors: Peiqi Yu, Nam Ling, Wei Wang, Wei Jiang

Organizations: School of Engineering, Santa Clara University, Santa Clara, CA, USA · Futurewei Technologies, Inc., USA

Abstract

Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive bias or a single orthogonal rotation on the output side of the retained weight. These corrections leave its input singular frame unchanged and therefore limit how the retained weight can adapt after column removal. We propose COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight. The right rotation is optimized on a reduced Stiefel manifold, while singular values are rescaled using generalized cross-validation to select the regularization strength for each layer. COEC further tempers the calibration Gram matrix to reduce the dominance of high-energy activation directions and introduces an alignment penalty that preserves the geometric relation between adjacent attention projections.All components use second-order statistics from a small calibration set and require neither backpropagation through the LLM nor retraining of the model parameters. COEC is independent of the column pruning criterion and can be applied to multiple structured pruning methods. Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families across multiple structured sparsity levels show that COEC improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher sparsity. These results show that post-pruning compensation can recover part of the performance lost to column removal.

Figures & tables

Figure 1: Overview of COEC. Columns are pruned using an arbitrary importance score, and the retained weight

W_{K}

is compensated as

\widetilde{W}=Q_{\ell}\,W_{K}^{(S)}\,Q_{r}

. The left and right rotations adjust the output and input singular frames, while per-mode rescaling adapts the singular values. All updates use calibration statistics only, without LLM backpropagation or model retraining. The objective combines relative reconstruction error with an alignment penalty on the attention output projection.

Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	14.35	6.20	13.03	15.15	13.90	6.02
Wanda-sp	COEC	10.89	5.61	9.09	13.26	9.64	5.17
FLAP	bias	11.89	5.86	8.65	8.82	6.85	5.29
FLAP	COEC	10.86	5.58	8.34	7.98	6.71	5.25
RCPU	RCPU	11.57	5.69	8.98	7.98	7.03	5.23
RCPU	COEC	10.91	5.58	8.91	7.63	6.74	5.21

Table 1: WikiText-2 perplexity

(\downarrow)

at 30% column sparsity. Each selection score is paired with its native compensation and with COEC under the same selection score and calibration data. Bold marks the better compensation within each selection, per model. Results at 10% and 20% sparsity are in App. A (supplementary material).

Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	47.4	68.7	49.1	53.7	57.5	71.1
Wanda-sp	COEC	48.9	70.4	56.0	53.8	61.0	72.5
FLAP	bias	44.8	68.8	52.6	55.9	65.6	70.5
FLAP	COEC	47.1	71.7	54.5	59.5	65.3	72.5
RCPU	RCPU	47.0	71.3	54.4	59.2	63.8	71.7
RCPU	COEC	50.2	71.7	56.1	58.5	64.2	72.0

Table 2: 7-task zero-shot accuracy %

(\uparrow)

at 30% column sparsity. Same protocol and bolding as Table 1 . Results at 10% and 20% sparsity are in App. A (supplementary material).

Figure 2: Perplexity versus calibration size on Llama-3.1-8B under the RCPU selection score. The selection is recomputed at each $N$ , so the prune-only baseline also varies.

Configuration	PPL $\downarrow$	ZS $\uparrow$
Prune $+$ RCPU compensation	10.06	50.9
$+$ GCV per-mode rescale	9.81	51.3
$+$ converged two-sided rotation	9.45	51.5
$+$ reduced-Stiefel right solve	9.86	52.7
$+$ Gram tempering ( $\alpha{=}0.9$ )	9.65	52.7
$+$ anchored align ( $\lambda_{a}{=}50$ )	9.65	52.9

Table 3: Component-wise build-up of COEC (last row), averaged over five development models at 30% sparsity. All rows use the RCPU selection score; indented rows accumulate from top to bottom.

	Params (B)		Mem (GB)		GFLOPs/tok
Model	dense	$30\%$	dense	$30\%$	dense	$30\%$
Qwen2.5-7B	7.62	5.70	15.0	11.5	15.2	11.4
Llama-3-70B	70.6	50.6	132.5	95.7	141.1	101.2

Table 4: Deployment cost at 30% column sparsity. The compensation does not change the model shape, so the figures are identical for prune-only, RCPU, and COEC.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

10% column sparsity
Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	7.32	3.87	6.87	7.04	6.33	4.17
Wanda-sp	COEC	6.86	3.72	6.78	6.06	5.43	4.09
FLAP	bias	6.89	3.50	6.77	6.12	5.24	4.01
FLAP	COEC	7.41	3.71	6.82	6.07	5.41	4.09
RCPU	RCPU	6.93	3.75	6.77	6.20	5.58	4.07

Appendix

Table 1: WikiText-2 perplexity

(\downarrow)

at 10% and 20% column sparsity. Protocol and bolding as Table 1 of the main paper.

10% column sparsity
Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	61.4	73.4	69.3	68.4	68.3	73.3
Wanda-sp	COEC	63.6	73.5	69.7	69.1	70.0	74.3
FLAP	bias	59.9	73.0	67.8	69.0	72.1	74.8
FLAP	COEC	60.7	74.0	70.0	67.5	71.2	74.5
RCPU	RCPU	62.6	74.0	69.1	65.3	71.2	74.5

Appendix

Table 2: 7-task zero-shot accuracy %

(\uparrow)

at 10% and 20% column sparsity. Protocol and bolding as Table 1 of the main paper.

10% column sparsity
Selection	Comp.	Llama-2-7B	Llama-2-13B	Llama-2-70B	Qwen3-8B	Qwen3-14B	Qwen3-32B
Wanda-sp	–	5.65	5.10	3.45	9.89	8.99	7.62
Wanda-sp	COEC	5.43	4.85	3.38	9.68	8.51	7.47
FLAP	bias	5.64	5.05	3.28	10.02	8.48	7.27
FLAP	COEC	5.54	4.99	3.26	9.90	8.37	7.22
RCPU	RCPU	5.43	4.87	3.38	9.97	8.51	7.52

Appendix

Table 3: WikiText-2 perplexity

(\downarrow)

on the Llama-2 and Qwen3 families across 10–30% column sparsity. Same protocol and bolding as Table 1 of the main paper.

10% column sparsity
Selection	Comp.	Llama-2-7B	Llama-2-13B	Llama-2-70B	Qwen3-8B	Qwen3-14B	Qwen3-32B
Wanda-sp	–	62.3	67.3	71.6	64.7	70.3	71.5
Wanda-sp	COEC	63.0	67.8	72.1	66.7	71.2	71.3
FLAP	bias	58.8	63.0	71.3	63.3	68.9	59.0
FLAP	COEC	58.6	62.8	71.0	63.4	69.1	72.2
RCPU	RCPU	62.5	67.4	71.3	67.8	70.8	71.4

Appendix

Table 4: 7-task zero-shot accuracy %

(\uparrow)

on the Llama-2 and Qwen3 families across 10–30% column sparsity. Same protocol and bolding as Table 1 of the main paper.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Sep 10, 2026

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (Latent-Informed Layer Analysis) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~~pp in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22

\times

reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Sankar Behera, Dhruv Singh, Anshika Agnihotri +3

CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1

LLM Pruning LLM Compression Structured Pruning Neural Network Pruning

Ask PDF

Jul 20, 2026

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.

Zhiren Gong, Zihao Zeng, Zijie Wang +3

College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

LLM Pruning LLM Compression Structured Pruning Attention Head Pruning Neural Network Pruning Transformer Pruning

Ask PDF

Aug 23, 2026

Correlation-Aware Structured Pruning for Large Language Models

Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Many existing methods assess the importance of prunable units (e.g., channels or heads) in isolation, implicitly assuming that pruning errors are additive. This independence assumption is often invalidated by the non-orthogonality of model weights and strong correlations between unit activations, potentially leading to performance degradation. To address this, we propose a Correlation-Aware Structured Pruning method. We formulate the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.

Sicheng Xu, Hao Shi, Wei Zhang +6

Department of Mathematical Sciences, Tsinghua University · Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co. Ltd.

LLM Pruning Neural Network Pruning Structured Sparsity

Ask PDF

Abstract

LLM Pruning

Explore similar work

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Correlation-Aware Structured Pruning for Large Language Models