Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

Kasun Dewage; Marianna Pensky; Heranga K. Rathnasekara; Suranadi De Silva

cs.CLAug 11, 2026

Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

Authors: Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara, Suranadi De Silva

Organizations: University of Central Florida Orlando, Florida, USA · Old Dominion University Norfolk, Virginia, USA

Abstract

Structured pruning of attention heads provides a hardware-friendly way to compress Transformer language models. However, existing methods for measuring head-level importance require calibration data, gradient computation, or Hessian estimation. These requirements add extra overhead and make the methods depend on the data. Our work presents Magnitude Profile (MP) scoring, a training-free criterion for head importance that identifies dispensable heads through statistical outlier detection on weight row norms. Heads whose projection weights fall within the population bulk are pruned, while heads exhibiting outlier norms, which carry disproportionate representational capacity, are preserved. Our work further gives MP-G, a variant that handles Grouped Query Attention (GQA) by distributing shared key-value group scores across associated query heads. Across five models evaluated on WikiText-2 perplexity at 12.5%-50% head sparsity, MP-G achieves the best perplexity on OPT-6.7B at all sparsity levels (18.46 at 12.5%, 27.87 at 25%, 152.0 at 50%). MP-G also gives the best results on RoBERTa-large at 12.5% and 25% sparsity, with perplexity values of 7.27 and 10.28, outperforming calibration-dependent baselines including Wanda-Head, SparseGPT-Head, and Gradient-Head. It requires zero forward passes, calibration samples, or gradient computation. At 50% sparsity, head pruning yields up to 16% parameter reduction with 50% attention FLOP savings. Our results show that weight-only statistical scoring can match or outperform data-dependent methods for structured head pruning, providing a practical, zero-cost criterion for Transformer compression.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Magnitude Profile Pruning (MP-G). (1) For each layer, L2 row norms are computed for the $W_{Q},W_{K},W_{V}$ projections and column norms for $W_{O}$ , using model weights only. (2) Per projection, the excess of each norm beyond the statistical threshold $\mu_{R}+z\sigma_{R}$ (Eq. 1 , $z{=}2.0$ by default) identifies outlier rows; norms within the population bulk receive zero excess. (3) Head scores aggregate query-side excesses ( $\mathrm{exc}_{Q}+\mathrm{exc}_{O}$ ) and, for GQA models, distribute the shared KV-group excess equally across the $g=H/H_{kv}$ query heads in the group (Eq. 3 ); for MHA models MP-G reduces to standard MP. (4) Heads are ranked globally by ascending score and the $\lfloor s\cdot L\cdot H\rfloor$ lowest-scoring heads are pruned by zeroing their $W_{Q}$ and $W_{O}$ slices, with $W_{K},W_{V}$ zeroed once all query heads in a KV group are pruned (Algorithm 1 ). The entire procedure requires no calibration samples, forward passes, gradients, or Hessian estimation.

	RoBERTa-base			RoBERTa-large			OPT-6.7B			LLaMA-3-8B			Mistral-7B
Method	12.5	25	50	12.5	25	50	12.5	25	50	12.5	25	50	12.5	25	50
Grad	8.22	11.8	64.7	7.79	81.8	13485	28.5	–	–	11.1	17.7	47.9	93.6	13.3	54.1
Wanda	8.89	16.5	103.8	8.01	13.7	86.0	42.5	152.3	406.3	1608	–	20009	107.0	472.6	552.8
SpGPT	11.2	113.4	4730	7.90	10.3	42.2	257.9	479.0	2034	142.3	344.6	1870	75.3	430.8	871.8
MP-G	10.0	17.8	71.3	7.27	10.3	45.8	18.5	27.9	152.0	20.4	41.4	7227	22.5	121.0	1072
Hybrid	9.09	14.0	96.1	7.82	11.1	85.6	25.6	136.3	499.6	40.3	467.9	13428	14.3	1290	4339

TABLE I: Perplexity (

\downarrow

) on WikiText-2 at head sparsity

s

with

z{=}2.0

. RoBERTa models report pseudo-perplexity (FP32); decoder models report causal LM perplexity (FP16). “–” indicates unavailable result. Best in bold , second underlined .

	$s{=}25\%$		$s{=}50\%$
Model	$z{=}1.5$	$z{=}2.0$	$z{=}2.0$	$z{=}2.5$
RoBERTa-base	10.67	17.79	71.25	73.67
RoBERTa-large	10.45	10.28	45.76	41.19
LLaMA-3-8B	54.98	41.41	7227	282.8
Mistral-7B	40.09	121.0	1072	2479

TABLE II: Effect of threshold

z

on MP-G perplexity at 25% and 50% sparsity.

	SST-2 (Acc)		MRPC (F1)		RTE (Acc)
Method	25%	50%	25%	50%	25%	50%
Wanda	93.5	81.4	28.8	0.0	51.6	47.3
SpGPT	94.6	89.6	88.6	81.4	75.1	47.3
MP-G	92.8	82.9	79.7	0.0	48.4	47.3
Hybrid	95.2	86.0	60.4	0.0	50.9	47.3

TABLE III: GLUE results for RoBERTa-large (fine-tune-then-prune). SST-2 and RTE: accuracy; MRPC: F1.

Method	Best	2nd	Calib. Required
Grad	8	1	Yes (backward)
MP-G	5	6	No
SpGPT	1	2	Yes (Hessian)
Hybrid	1	3	Yes (forward)
Wanda	0	3	Yes (forward)

TABLE IV: Number of model-sparsity pairs where each method achieves best or second-best perplexity (out of 15 comparable pairs at

z{=}2.0

with

{\geq}2

finite results).

Explore similar work

Complementary Attention Head Pruning for Efficient Transformers
Jun 17, 2026Yaniv Livertovsky, Shahar Somin, Gonen SingerModel Compression Transformer Attention
Bar-Ilan University Ramat Gan, Israel
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Mar 22, 2026Abhinaba Basu, Kumkum Basu, Koushik DebLLM Compression LLM Pruning
Indian Institute of Information Technology Allahabad, India · National Institute of Electronics and Information Technology (NIELIT), India · Indian Institute of Technology Patna, India +2

Jun 17, 2026J/K move · Enter open · S save

Complementary Attention Head Pruning for Efficient Transformers

Yaniv Livertovsky, Shahar Somin, Gonen Singer

Bar-Ilan University Ramat Gan, Israel

Model Compression Transformer Attention Attention Head Pruning Neural Network Pruning

Open paper AskPDF arXiv

The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments. While structured pruning offers a pathway to compression, existing state-of-the-art methods often rely on gradient-based importance ranking or stochastic gating, which suffer from instability, structural degeneration, and the need for extensive manual hyperparameter tuning. In this paper, we introduce CAHP (Complementary Attention Head Pruning), a novel post-hoc framework that redefines head selection as a global graph-theoretical problem. Rather than evaluating heads in isolation, CAHP utilizes graph-based clustering combined with information-theoretic distance measures to identify and preserve a topologically diverse subset of complementary attention heads. Without requiring a predefined sparsity level or pruning ratio, the framework automatically determines the number of selected attention heads across layers by identifying a diminishing marginal performance curve, where pruning additional heads leads to a sharp degradation in performance, as determined by the chosen polynomial degree. Extensive evaluations on the SST-5 and MNLI benchmarks, across different Transformer model scales, demonstrate that CAHP consistently outperforms competitive baselines, particularly in high-compression regimes. Furthermore, our structural analysis shows that CAHP avoids the "proximity bias" of gradient-based pruning methods, which tend to preserve heads mainly in layers close to the output, and instead retains a functionally critical set of attention heads in the model's intermediate layers.

Abstract

Explore similar work

Complementary Attention Head Pruning for Efficient Transformers

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal