COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

Peiqi Yu; Nam Ling; Wei Wang; Wei Jiang

cs.LGAug 21, 2026

COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

Authors: Peiqi Yu, Nam Ling, Wei Wang, Wei Jiang

Organizations: School of Engineering, Santa Clara University, Santa Clara, CA, USA · Futurewei Technologies, Inc., USA

Abstract

Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive bias or a single orthogonal rotation on the output side of the retained weight. These corrections leave its input singular frame unchanged and therefore limit how the retained weight can adapt after column removal. We propose COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight. The right rotation is optimized on a reduced Stiefel manifold, while singular values are rescaled using generalized cross-validation to select the regularization strength for each layer. COEC further tempers the calibration Gram matrix to reduce the dominance of high-energy activation directions and introduces an alignment penalty that preserves the geometric relation between adjacent attention projections.All components use second-order statistics from a small calibration set and require neither backpropagation through the LLM nor retraining of the model parameters. COEC is independent of the column pruning criterion and can be applied to multiple structured pruning methods. Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families across multiple structured sparsity levels show that COEC improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher sparsity. These results show that post-pruning compensation can recover part of the performance lost to column removal.

Figures & tables

Figure 1: Overview of COEC. Columns are pruned using an arbitrary importance score, and the retained weight

W_{K}

is compensated as

\widetilde{W}=Q_{\ell}\,W_{K}^{(S)}\,Q_{r}

. The left and right rotations adjust the output and input singular frames, while per-mode rescaling adapts the singular values. All updates use calibration statistics only, without LLM backpropagation or model retraining. The objective combines relative reconstruction error with an alignment penalty on the attention output projection.

Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	14.35	6.20	13.03	15.15	13.90	6.02
Wanda-sp	COEC	10.89	5.61	9.09	13.26	9.64	5.17
FLAP	bias	11.89	5.86	8.65	8.82	6.85	5.29
FLAP	COEC	10.86	5.58	8.34	7.98	6.71	5.25
RCPU	RCPU	11.57	5.69	8.98	7.98	7.03	5.23
RCPU	COEC	10.91	5.58	8.91	7.63	6.74	5.21

Table 1: WikiText-2 perplexity

(\downarrow)

at 30% column sparsity. Each selection score is paired with its native compensation and with COEC under the same selection score and calibration data. Bold marks the better compensation within each selection, per model. Results at 10% and 20% sparsity are in App. A (supplementary material).

Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	47.4	68.7	49.1	53.7	57.5	71.1
Wanda-sp	COEC	48.9	70.4	56.0	53.8	61.0	72.5
FLAP	bias	44.8	68.8	52.6	55.9	65.6	70.5
FLAP	COEC	47.1	71.7	54.5	59.5	65.3	72.5
RCPU	RCPU	47.0	71.3	54.4	59.2	63.8	71.7
RCPU	COEC	50.2	71.7	56.1	58.5	64.2	72.0

Table 2: 7-task zero-shot accuracy %

(\uparrow)

at 30% column sparsity. Same protocol and bolding as Table 1 . Results at 10% and 20% sparsity are in App. A (supplementary material).

Figure 2: Perplexity versus calibration size on Llama-3.1-8B under the RCPU selection score. The selection is recomputed at each $N$ , so the prune-only baseline also varies.

Configuration	PPL $\downarrow$	ZS $\uparrow$
Prune $+$ RCPU compensation	10.06	50.9
$+$ GCV per-mode rescale	9.81	51.3
$+$ converged two-sided rotation	9.45	51.5
$+$ reduced-Stiefel right solve	9.86	52.7
$+$ Gram tempering ( $\alpha{=}0.9$ )	9.65	52.7
$+$ anchored align ( $\lambda_{a}{=}50$ )	9.65	52.9

Table 3: Component-wise build-up of COEC (last row), averaged over five development models at 30% sparsity. All rows use the RCPU selection score; indented rows accumulate from top to bottom.

	Params (B)		Mem (GB)		GFLOPs/tok
Model	dense	$30\%$	dense	$30\%$	dense	$30\%$
Qwen2.5-7B	7.62	5.70	15.0	11.5	15.2	11.4
Llama-3-70B	70.6	50.6	132.5	95.7	141.1	101.2

Table 4: Deployment cost at 30% column sparsity. The compensation does not change the model shape, so the figures are identical for prune-only, RCPU, and COEC.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

10% column sparsity
Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	7.32	3.87	6.87	7.04	6.33	4.17
Wanda-sp	COEC	6.86	3.72	6.78	6.06	5.43	4.09
FLAP	bias	6.89	3.50	6.77	6.12	5.24	4.01
FLAP	COEC	7.41	3.71	6.82	6.07	5.41	4.09
RCPU	RCPU	6.93	3.75	6.77	6.20	5.58	4.07

Appendix

Table 1: WikiText-2 perplexity

(\downarrow)

at 10% and 20% column sparsity. Protocol and bolding as Table 1 of the main paper.

10% column sparsity
Selection	Comp.	Llama-3.1-8B	Llama-3-70B	Qwen2.5-7B	Qwen2.5-14B	Qwen2.5-32B	Qwen2.5-72B
Wanda-sp	–	61.4	73.4	69.3	68.4	68.3	73.3
Wanda-sp	COEC	63.6	73.5	69.7	69.1	70.0	74.3
FLAP	bias	59.9	73.0	67.8	69.0	72.1	74.8
FLAP	COEC	60.7	74.0	70.0	67.5	71.2	74.5
RCPU	RCPU	62.6	74.0	69.1	65.3	71.2	74.5

Appendix

Table 2: 7-task zero-shot accuracy %

(\uparrow)

at 10% and 20% column sparsity. Protocol and bolding as Table 1 of the main paper.

10% column sparsity
Selection	Comp.	Llama-2-7B	Llama-2-13B	Llama-2-70B	Qwen3-8B	Qwen3-14B	Qwen3-32B
Wanda-sp	–	5.65	5.10	3.45	9.89	8.99	7.62
Wanda-sp	COEC	5.43	4.85	3.38	9.68	8.51	7.47
FLAP	bias	5.64	5.05	3.28	10.02	8.48	7.27
FLAP	COEC	5.54	4.99	3.26	9.90	8.37	7.22
RCPU	RCPU	5.43	4.87	3.38	9.97	8.51	7.52

Appendix

Table 3: WikiText-2 perplexity

(\downarrow)

on the Llama-2 and Qwen3 families across 10–30% column sparsity. Same protocol and bolding as Table 1 of the main paper.

10% column sparsity
Selection	Comp.	Llama-2-7B	Llama-2-13B	Llama-2-70B	Qwen3-8B	Qwen3-14B	Qwen3-32B
Wanda-sp	–	62.3	67.3	71.6	64.7	70.3	71.5
Wanda-sp	COEC	63.0	67.8	72.1	66.7	71.2	71.3
FLAP	bias	58.8	63.0	71.3	63.3	68.9	59.0
FLAP	COEC	58.6	62.8	71.0	63.4	69.1	72.2
RCPU	RCPU	62.5	67.4	71.3	67.8	70.8	71.4

Appendix

Table 4: 7-task zero-shot accuracy %

(\uparrow)

on the Llama-2 and Qwen3 families across 10–30% column sparsity. Same protocol and bolding as Table 1 of the main paper.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Sep 10, 2026Sankar Behera, Dhruv Singh, Anshika Agnihotri +3LLM Pruning LLM Compression
CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1
CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
Jul 20, 2026Zhiren Gong, Zihao Zeng, Zijie Wang +3LLM Pruning LLM Compression
College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
Correlation-Aware Structured Pruning for Large Language Models
Aug 23, 2026Sicheng Xu, Hao Shi, Wei Zhang +6LLM Pruning Neural Network Pruning
Department of Mathematical Sciences, Tsinghua University · Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co. Ltd.

Sep 10, 2026J/K move · Enter open · S save

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary+2

CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner+1

LLM Pruning LLM Compression Structured Pruning Neural Network Pruning

Open paper AskPDF arXiv

Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (Latent-Informed Layer Analysis) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~~pp in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22

\times

reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Abstract

LLM Pruning

Explore similar work

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Correlation-Aware Structured Pruning for Large Language Models