LLM Pruning

LLM: Large Language Model

Large language model pruning studies how to remove weights, neurons, attention heads, layers, or experts while retaining useful behavior. Methods range from magnitude and activation-based scoring to gradient, curvature, causal, and representation-based criteria, often using calibration data to estimate which components matter; some also recover performance through fine-tuning or distillation. Structured pruning can reduce computation more directly, while unstructured sparsity often needs specialized hardware and kernels to deliver real speedups. A central challenge is that the best pruning decisions depend on the model, task, calibration data, and deployment setting, and preserving benchmark accuracy may not preserve generation quality, fairness, or reliability.

Latest papers

178 papers32 papers in the last four weeks, up 191% on the four weeks before. 0.2% of all new papers.

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models
Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLLM Pruning Attention Head Pruning
Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka
Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning
Oct 25, 2026Tao Jing, Chen Kang, Feifei Zhai +2Efficient Language Model Reasoning LLM Pruning
T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding
Oct 25, 2026Shahriar Kabir Nahin, Xiao Ye, Muhao Chen +2Speculative Decoding LLM Inference Acceleration
Bellini College of AI, Cybersecurity, and Computing, University of South Florida · School of Computing and Augmented Intelligence, Arizona State University · Department of Computer Science, University of California, Davis
Direct Pass@ $k$ Optimization for Thought Pruning: Balancing Quality and Diversity in Test-Time Reasoning
Oct 22, 2026Mahdi Salmani, Adel Javanmard, Devansh Gupta +1Efficient Language Model Reasoning LLM Pruning
University of Southern California
Building a Nested LADDER for LLM Unstructured Post-Training Pruning
Oct 20, 2026Nicolás Mauricio Cuadrado Ávila, Ivo Gollini Navarrete, Martin Takáč +1LLM Pruning Neural Network Pruning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
Oct 20, 2026Irene Tenison, Abdulrahman Alabdulkareem, Daniel Karl I. Weidele +2KV-Cache Compression Transformer Attention
MIT CSAIL · MIT-IBM Computing Research Lab
ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models
Oct 20, 2026Jiahui Ning, Liang Chen, Yuchen Liang +2LLM Compression LLM Evaluation
City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute +1
SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning
Oct 20, 2026Guanchen Li, Yixing Xu, Xuanwu Yin +2LLM Pruning Transformer Pruning
Advanced Micro Devices, Inc. (AMD)
How Fine-Tuning Objectives Shape Layer-Wise Information in LLM Hallucination Detection
Oct 20, 2026Kaiyang Wan, Forrest Sheng Bao, Amin Ahmad +1Fine-Tuning Language Models LLM Hallucination Detection
INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · PAIR Systems, Inc., Mountain View, California
LOOP: Recovery-Aligned Iterative Layer Pruning via On-Policy Distillation
Oct 19, 2026Liang Xu, Shufan Shen, Zhongni Hou +4LLM Pruning Neural Network Pruning
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan
Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning
Oct 16, 2026Mohamed Shaaban, Mohamed ElmahallawyPrivacy Leakage in Language Models Privacy-Preserving Language Models
School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA
Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models
Oct 15, 2026Vedant Pandhare, Piyush DhokaLLM Pruning Neural Network Pruning
Independent Researcher
To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models
Oct 14, 2026Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu +2Repetition in Language Generation LLM Reasoning
Nanyang Technological University, Singapore · VinUniversity, Vietnam
Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
Oct 13, 2026Woosang Lim, Haneul Yoo, Vincent-Daniel Yun +1LLM Pruning LLM Compression
Seoul National University, Republic of Korea · New York University, United States · University of Southern California, United States
Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
Oct 13, 2026Mehdi Makni, Natalia Ponomareva, Rahul MazumderLLM Pruning Neural Network Pruning
Operations Research Center, Massachusetts Institute of Technology · Google Research
From Spectral Saliency to Finite Deletion: Evaluating Selective Language Model Edits
Oct 13, 2026Ibne Farabi Shihab, Joyanta Jyoti MondalLanguage Model Editing LLM Pruning
Department of Computer and Information Sciences, University of Delaware, USA
Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models
Oct 11, 2026Cameron Barker, Andreas Paraskeva, Jan N. van Rijn +2LLM Pruning Neural Network Pruning
School of Informatics, University of Edinburgh, UK · Leiden Institute of Advanced Computer Science, Leiden University, the Netherlands · School of Engineering, University of Edinburgh, UK
The Fragility and Dynamics of LLM Pruning-Recovery Steps
Oct 11, 2026Luis Frentzen Salim, Komang Rinartha, Chia-Chun Wu +3LLM Pruning LLM Compression
Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute
SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning
Oct 11, 2026Sankar Behera, Dhruv Singh, Satyadev Ahlawat +1LLM Pruning Structured Pruning
Department of Computer Science and Engineering, IIT Jammu · Department of Mathematics, IIT Jammu · Department of Electrical Engineering, IIT Jammu
WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning
Oct 11, 2026Shengkun Tang, Dan Alistarh, Zhiqiang ShenLLM Pruning Structured Pruning
MBZUAI · ISTA
Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models
Oct 10, 2026Muhammad Usama, Yeoreum JungLLM Compression Language Model Merging
Braindeck Inc Seoul, Republic of Korea
EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model
Oct 10, 2026S M Mahbub Murshed, Hayden Bright, Rezaul Karim +1KV-Cache Compression KV-Cache Quantization
Causal Dynamics Lab · Electrical Engineering and Computer Science Department, University of Missouri-Columbia · Computer Science & Engineering, University of Chittagong
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Oct 8, 2026Qitong Wang, Xinwei Niu, Mingluo Su +3LLM Pruning LLM Inference Acceleration
Westlake University · Ant Group
Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance
Oct 8, 2026Zheng Huang, Sansheng Cao, Enpei Zhang +7LLM Reasoning LLM Interpretability
Dartmouth College · Yale University · New York University +2
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
Oct 8, 2026Houcheng Jiang, Mao Zheng, Mingyang Song +4LLM Pruning LLM Reasoning
Zhongguancun Academy · Foundation Model Department, Tencent · University of Science and Technology of China +1
TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery
Oct 6, 2026Yuanzhe Li, Pengxin Wang, Yuxin Ren +5LLM Pruning LLM Inference Acceleration
University of Arizona · University of Pittsburgh · University of Central Florida
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
Oct 6, 2026Davis Wertheimer, Haochen Shen, Ahan Gupta +6KV-Cache Compression Transformer Attention
IBM Research, USA · SSAIL Lab, University of Illinois Urbana-Champaign
Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compression
Oct 5, 2026Hankyul Kang, Jongbin RyuLLM Quantization LLM Pruning
Ajou University
Capability Scaling-Down Laws for LLM Compression
Oct 1, 2026Xueqi Cheng, Liang Wu, Kelly Wan +2LLM Compression LLM Pruning
Florida State University · Nokia Applied Research
QK-Wanda: Coupling Queries and Keys for Unstructured Pruning
Oct 1, 2026Ivan Ilin, Peter RichtárikLLM Pruning Neural Network Pruning
KAUST

Oct 27, 2026J/K move · Enter open · S save

Mechanistic Failure Modes of Weight Pruning in Transformer Language Models

Nabeelah Faumi, Deshan Sumanathilaka

Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

LLM Pruning Attention Head Pruning LLM Interpretability Circuits in Language Models

Open paper AskPDF arXiv

Pruned language models are judged almost entirely by accuracy and perplexity, which assumes that preserved output behaviour implies preserved internal computation. We test this assumption with a factorial pruning study on GPT-2, crossing two importance criteria (magnitude, random), two component types (attention heads, MLP neurons) and three sparsity levels (3%, 50%, 70%). Using the fully characterised indirect object identification (IOI) circuit as ground truth, we measure circuit integrity with activation patching, path patching and direct logit attribution. We find two qualitatively distinct failure modes. Magnitude head pruning >= zeroes the circuit: output-facing heads are removed, and the model falls to near-chance. Magnitude neuron pruning at 30% inverts it: all 26 circuit heads remain intact, but Name Mover contributions flip sign, and the model systematically prefers the wrong name (17.7% accuracy). Perplexity ranks it as less damaged than random head pruning (74.6 vs 140.1). A replication of the Greater-Than circuit shows that the critical component type depends on the circuit's architecture. These results show that output metrics and structural checks can both miss mechanistic damage, motivating circuit-level causal analysis as a complementary evaluation of compressed language models.