WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

Shengkun Tang; Dan Alistarh; Zhiqiang Shen

cs.LGOct 11, 2026

WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

Authors: Shengkun Tang, Dan Alistarh, Zhiqiang Shen

Organizations: MBZUAI · ISTA

Abstract

Structured pruning is typically formulated as identifying which model dimensions to remove, often within the model's original representation. However, the effect of coordinate deletion fundamentally depends on the basis in which pruning is performed. This raises a more fundamental question: how should the basis be chosen specifically for coordinate deletion? We introduce WUSPruner, a transform-based framework for structured pruning of large language models. We formulate basis selection as a local linear reconstruction problem that directly minimizes the reconstruction loss induced by pruning. Inspired by the second-moment transformation in WUSH, we consider a WUS transform constructed jointly from activation and weight statistics. We show that under positive-definite second moments, WUS transform minimizes the local reconstruction loss over all invertible changes of basis. In the resulting basis, the activation and weight second moments are simultaneously diagonalized and balanced, causing the deletion error to decompose into independent spectral contributions and yielding a principled criterion for coordinate selection. Guided by this result, we develop a scalable pruning method based on blockwise WUS transformations and compact input projections. Experiments on Llama and Qwen models from 8B to 72B parameters show consistent improvements over structured pruning baselines across language modeling, knowledge, commonsense reasoning, mathematics, and code generation benchmarks, while also delivering practical reductions in inference latency and memory usage.

Figures & tables

Figure 1: Overview of WUSPruner . WUS constructs a basis from activation and weight second moments such that the transformed moments are balanced and diagonalized with a shared spectrum $S$ . This makes the pruning loss decompose into independent coordinate-wise contributions, allowing transformed coordinates to be ranked by $s_{i}$ and the least important matched dimensions to be structurally removed.

Model	Method	Zero-shot Acc. $\uparrow$							Perplexity $\downarrow$
Model	Method	PIQA	HellaSwag	ARC-C	MMLU	OBQA	CMMLU	Avg.	Wiki2	PTB	C4	Avg.
Llama3-8B	Dense	80.3	79.2	53.6	61.9	45.0	48.5	61.4	6.1	9.9	9.3	8.4
	Wanda-sp	65.2	34.5	28.8	26.5	29.8	25.1	34.9	253.6	336.7	313.7	301.3
	SliceGPT	62.8	49.6	31.4	25.0	34.0	25.3	38.0	12.1	116.1	121.7	83.3
	WUSPruner (Ours)	69.2	53.8	32.5	37.2	33.0	29.4	42.5	14.8	24.8	31.3	23.6
Qwen3-8B	Dense	77.8	74.7	56.5	72.9	42.6	76.9	66.9	9.7	15.4	15.2	13.4

Table 1: Main results on commonsense reasoning, knowledge, and language modeling benchmarks across three LLMs with 25% compression ratio. We report zero-shot accuracy (%) and perplexity (PPL), where higher accuracy and lower perplexity indicate better performance. Bold values denote the best results among pruning methods, excluding the dense model.

Method	Zero-shot Acc. $\uparrow$
Method	HumanEval	HumanEval-Plus	MBPP	MBPP-Plus	GSM-8K	GSM-Platinum	Avg.
Dense	84.8	79.9	78.6	63.8	87.4	70.6	77.5
Wanda-sp	4.9	4.3	11.1	9.5	4.2	3.1	6.2
SliceGPT	62.8	55.5	54.2	44.1	56.4	19.3	48.7
WUSPruner (Ours)	64.0	57.3	57.1	49.5	58.9	37.3	54.0

Table 2: Performance on mathematical reasoning and code generation benchmarks using Qwen3.5-9B with 25% compression ratio. We report accuracy (%) across six benchmarks and their average.

Model	Method	Zero-shot Acc. $\uparrow$							Perplexity $\downarrow$
Model	Method	PIQA	HellaSwag	ARC-C	MMLU	OBQA	CMMLU	Avg.	Wiki2	PTB	C4	Avg.
Qwen3.8-27B	Dense	81.6	82.9	58.5	83.4	46.2	71.7	70.7	7.0	8.8	11.0	8.9
	Wanda-sp	71.6	66.1	37.3	55.4	37.2	43.9	51.9	24.5	24.2	25.7	24.8
	SliceGPT	70.1	63.7	43.0	37.2	40.2	28.6	47.1	8.7	29.3	25.9	21.3
	WUSPruner (Ours)	75.7	68.6	51.1	69.1	40.6	60.5	60.9	9.1	13.3	16.9	13.1
Qwen2.5-72B	Dense	83.6	86.0	62.7	83.3	46.2	89.9	75.2	3.8	8.4	9.1	7.1

Table 3: Evaluation of WUSPruner on larger language models, including Qwen3.8-27B and Qwen2.5-72B with 25% compression ratio.

Panel of Figure 2 — Figure 2: Sensitivity analysis of WUSPruner with respect to block size, calibration set size, and pruning ratio on Qwen3-8B and Qwen3.5-9B. Larger block sizes generally yield lower perplexity, consistent with a closer approximation to the original WUS transformation, while increasing computational overhead. WUSPruner remains stable across a wide range of calibration set sizes, and its performance degrades smoothly as the pruning ratio increases.

Table 6

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description	Shape / Domain
$X$	Calibration activations	$\mathbb{R}^{n\times d}$
$W$	Weight matrix of a linear layer	$\mathbb{R}^{d\times m}$
$d$	Original input dimension	–
$r$	Number of retained coordinates	–
$T$	Invertible change-of-basis transformation	$\mathrm{GL}(d)$
$P$	Diagonal projector selecting deleted coordinates	$\mathbb{R}^{d\times d}$

Appendix

Table 6: Summary of the main notation used in the paper.

Method	Zero-shot Acc. $\uparrow$							Perplexity $\downarrow$
Method	PIQA	HellaSwag	ARC-C	MMLU	OBQA	CMMLU	Avg.	Wiki2	PTB	C4	Avg.
Dense	80.3	78.0	55.8	78.6	42.8	80.3	69.3	8.6	12.9	11.9	11.1
Wanda-sp	67.4	43.5	33.1	45.0	31.0	41.0	43.5	73.5	74.7	67.1	71.8
SliceGPT	58.3	37.8	27.2	32.5	27.4	29.0	35.4	53.1	58.6	57.6	56.4
WUSPruner (Ours)	68.1	50.5	38.9	52.1	30.6	54.1	49.1	21.8	29.8	28.9	26.8

Appendix

Table 7: Additional evaluation of Qwen3.5-9B calibrated with the mixed mathematics and coding dataset at a 25% compression ratio.

Method	Performance $\uparrow$							Perplexity $\downarrow$
Method	PIQA	HellaSwag	ARC-C	MMLU	OBQA	CMMLU	Avg.	Wiki2	PTB	C4	Avg.
Dense	77.8	74.7	56.5	72.9	42.6	76.9	66.9	9.7	15.4	15.2	13.4
SVD-LLM	67.6	54.3	38.9	45.5	38.2	37.7	47.0	15.1	45.9	93.3	51.4
WUSPruner (Ours)	67.8	53.7	39.2	54.3	34.6	47.5	49.5	15.0	28.6	33.7	25.8

Appendix

Table 8: Additional comparison with SVD-LLM on Qwen3-8B at a 25% compression ratio.

Model	Sparsity	Zero-shot Acc. $\uparrow$							Perplexity $\downarrow$
Model	Sparsity	PIQA	HellaSwag	ARC-C	MMLU	OBQA	CMMLU	Avg.	Wiki2	PTB	C4	Avg.
Qwen3-8B	15%	74.1	64.7	50.6	64.8	39.6	64.1	59.5	12.0	19.3	20.9	17.4
	20%	71.2	59.4	43.5	60.2	37.6	56.3	54.7	13.5	22.8	25.2	20.5
	25%	67.8	53.7	39.2	54.3	34.6	47.5	49.5	15.0	28.6	33.7	25.8
	30%	63.0	47.8	33.1	47.7	30.6	36.8	43.1	17.5	41.2	44.3	34.3
	35%	60.7	42.3	28.2	37.7	30.6	29.9	38.2	21.4	59.1	64.3	48.2

Appendix

Table 9: Benchmark performance of WUSPruner under different compression ratios on Qwen3-8B and Qwen3.5-9B.

Method	Qwen3-8B	Qwen3.5-9B	Qwen3.8-27B	Qwen2.5-72B
Wanda-sp	0.2	0.3	0.8	1.1
WUSPruner (Ours)	0.5	0.67	1.5	2.3

Appendix

Table 10: Runtime comparison of different compression methods. We report the total wall-clock time required to compress each model using the same calibration setting.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Sep 10, 2026

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (Latent-Informed Layer Analysis) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~~pp in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22

\times

reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Sankar Behera, Dhruv Singh, Anshika Agnihotri +3

CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1

LLM Pruning LLM Compression Structured Pruning Neural Network Pruning

Ask PDF

Apr 21, 2026

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a single global budget. Instead of learning importance scores without constraints and applying the budget only after training, GRASPrune learns lightweight gate scores with a projected straight-through estimator that enforces a hard mask satisfying the budget at every step while keeping the backbone weights frozen. After the mask is fixed, we calibrate scaling factors on the retained units to mitigate scale mismatch caused by pruning, and fold these factors into the pruned weights to obtain a smaller dense checkpoint with no extra parameters at inference. On LLaMA-2-7B, GRASPrune removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five benchmarks, using four epochs on 512 unlabeled calibration sequences on a single NVIDIA A100 80GB GPU without any full model fine-tuning.

Ziyang Wang, Jiangfeng Xiao, Chuan Xiao +3

Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University

LLM Pruning LLM Compression Structured Pruning

Ask PDF

Jul 20, 2026

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.

Zhiren Gong, Zihao Zeng, Zijie Wang +3

College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

LLM Pruning LLM Compression Structured Pruning Attention Head Pruning Neural Network Pruning Transformer Pruning

Ask PDF

Abstract

LLM Pruning

Explore similar work

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning