Attention Head Pruning

Follow this topic Ask about this topic

Attention head pruning studies which heads in Transformer models can be removed while preserving useful behavior, making models cheaper to store and run. Methods score heads using weight or activation statistics, gradients and causal tests, redundancy or complementarity measures, and learned or input-dependent gates. Researchers apply these approaches to language models, vision Transformers, and multimodal systems, evaluating both task quality and practical measures such as latency and throughput. A central challenge is that a head’s importance can depend on the task or input, and theoretical sparsity does not always translate into real speedups unless the resulting structure works well with hardware and kernels.

Latest papers

22 papers4 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models
Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLLM Pruning Attention Head Pruning
Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka
Your Transformer Is Secretly Elastic
Oct 14, 2026Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts +1Elastic Neural Networks Transformer Pruning
University of Amsterdam · TNO, Intelligent Imaging
BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels
Oct 12, 2026Yueyuan Sui, Payal Mohapatra, Qi Zhu +1Dynamic Sparse Training Neural Network Pruning
Northwestern University, USA · Arm Inc., USA
Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance
Oct 8, 2026Zheng Huang, Sansheng Cao, Enpei Zhang +7LLM Reasoning LLM Interpretability
Dartmouth College · Yale University · New York University +2
CASS: Contribution-Aware Structured Sparsity for Model Merging
Sep 28, 2026Yan Li, Guiping Cao, Meng Xu +5Multi-Task Learning Structured Sparsity
Pengcheng Laboratory, Shenzhen, China · City University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2
Diffract: Spectral View of LLM Domain Adaptation
Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain Adaptation Domain-Adaptive Pretraining
Risk AI Research Lab · Applied AI Institute
Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression
Aug 11, 2026Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara +1LLM Compression Transformer Pruning
University of Central Florida Orlando, Florida, USA · Old Dominion University Norfolk, Virginia, USA
Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
Jul 31, 2026Kamil Książek, Piotr Suszyński, Michał Jan Włodarczyk +2Transformer Interpretability Attention Head Analysis
Centre for Credible Artificial Intelligence Warsaw University of Technology Warsaw, Poland · Faculty of Mathematics and Computer Science Jagiellonian University Krakow, Poland · University of Warsaw Warsaw, Poland
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Jul 30, 2026Haozhe Hu, Hao Wu, Peiran Yin +3LLM Pruning Token Pruning
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Munich Center for Machine Learning, LMU Munich
Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
Jul 21, 2026Maohua Li, Qirui Li, Yanke Zhou +10Transformer Interpretability Diffusion Transformer
Nanjing University · Alibaba Group · Zhejiang University
CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
Jul 20, 2026Zhiren Gong, Zihao Zeng, Zijie Wang +3LLM Pruning LLM Compression
College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
MobileWan: Closing the Quality Gap for Mobile Video Diffusion
Jul 7, 2026Mohsen Ghafoorian, Denis Korzhenkov, Adil Karjauv +9Video Diffusion Models Video Generation
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Complementary Attention Head Pruning for Efficient Transformers
Jun 17, 2026Yaniv Livertovsky, Shahar Somin, Gonen SingerModel Compression Transformer Attention
Bar-Ilan University Ramat Gan, Israel
From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment
May 20, 2026Hao Chen, Qi Zhang, Liyao Li +7LLM Alignment LLM Fine-Tuning
Zhejiang University · Ant Group · Eastern Institute of Technology
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Apr 27, 2026Amogh Sheth, Andrew Lin, Biruk Assefa +2LLM Pruning Self-Attention
Edison Academy Magnet School · Massachusetts Institute of Technology · State University of New York College at Plattsburgh +2
INCRT: An Incremental Transformer That Determines Its Own Architecture
Apr 12, 2026Giansalvo CirrincioneTransformer Neural Architecture Search
Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France
Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference
Apr 10, 2026Yueyuan Sui, Payal Mohapatra, Doğaç Eldenk +5Efficient Multimodal Inference Neural Network Pruning
Northwestern University, Evanston, Illinois, USA
Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)
Mar 23, 2026Yousung Lee, Dongsoo HarAttention Head Pruning Efficient ViTs
Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea
HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers
Mar 12, 2026Andy Li, Aiden Durrant, Milan Markovic +1Efficient ViTs Attention Head Pruning
School of Natural and Computing Sciences, University of Aberdeen, UK · School of Computing Sciences, University of East Anglia, UK · Department of Physics and Technology, UiT The Arctic University of Norway, Norway
ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
Feb 17, 2026Hyunchan Moon, Cheonjun Park, Steven L. WaslanderChannel Pruning Efficient ViTs
LG Electronics, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea · University of Toronto, Toronto, Canada
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University

Oct 27, 2026J/K move · Enter open · S save

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models

Nabeelah Faumi, Deshan Sumanathilaka

Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

LLM Pruning Attention Head Pruning LLM Interpretability Circuits in Language Models

Open paper AskPDF arXiv

Pruned language models are judged almost entirely by accuracy and perplexity, which assumes that preserved output behaviour implies preserved internal computation. We test this assumption with a factorial pruning study on GPT-2, crossing two importance criteria (magnitude, random), two component types (attention heads, MLP neurons) and three sparsity levels (3%, 50%, 70%). Using the fully characterised indirect object identification (IOI) circuit as ground truth, we measure circuit integrity with activation patching, path patching and direct logit attribution. We find two qualitatively distinct failure modes. Magnitude head pruning >= zeroes the circuit: output-facing heads are removed, and the model falls to near-chance. Magnitude neuron pruning at 30% inverts it: all 26 circuit heads remain intact, but Name Mover contributions flip sign, and the model systematically prefers the wrong name (17.7% accuracy). Perplexity ranks it as less damaged than random head pruning (74.6 vs 140.1). A replication of the Greater-Than circuit shows that the critical component type depends on the circuit's architecture. These results show that output metrics and structural checks can both miss mechanistic damage, motivating circuit-level causal analysis as a complementary evaluation of compressed language models.