LLM Pruning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 113

All topics

Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning
Jun 2, 2026Hu Xu, Zhaolong Xing, Congcong Liu +5LLM Pruning Model Calibration
Shanghai Jiao Tong University
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
Jun 1, 2026Elia Cunegatti, Marcus Vukojevic, Erik Nielsen +1LLM Pruning LLM Compression
University of Trento
Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search
Jun 1, 2026Cheonjun ParkLLM Pruning LLM Compression
Hankuk University of Foreign Studies
Before Parc Fermé: RL-Time Pruning for Efficient Embodied LLMs in Autonomous Driving
May 29, 2026Luca Benfenati, Ali Azimi, Matteo Risso +3LLM Pruning Language Model-Based Control
Department of Control and Computer Engineering, Politecnico di Torino · Department of Mechanical and Aerospace Engineering, Politecnico di Torino
AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
May 28, 2026Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan KandemirLLM Pruning Vision-Language Models
The Pennsylvania State University
Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
May 28, 2026Junlin He, Yihong Tang, Tong Nie +5LLM Pruning LLM Compression
The Hong Kong Polytechnic University, Hong Kong SAR, China · McGill University, Montreal, QC, Canada
Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules
May 27, 2026Karim Galliamov, Rochelle Choenni, Ivan TitovLLM Pruning LLM Compression
University of Amsterdam · University of Edinburgh
PrunePath: Towards Highly Structured Sparse Language Models
May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts Pruning LLM Pruning
Department of Applied Mathematics, The Hong Kong Polytechnic University
Pruning and Distilling Mixture-of-Experts into Dense Language Models
May 27, 2026Junhyuck Kim, Jihun Yun, Haechan Kim +3Mixture-of-Experts Pruning LLM Pruning
1KRAFTON · 2KAIST
Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
May 27, 2026Liu O. Martin, Lucas Bandarkar, Nanyun PengMixture-of-Experts Pruning LLM Pruning
University of California, Los Angeles
Locality-Aware Redundancy Pruning for LLM Depth Compression
May 27, 2026Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3LLM Pruning LLM Compression
University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
May 26, 2026Fengfa Li, Hongjin Ji, Yifeng Ding +2Mixture-of-Experts Pruning LLM Pruning
Li Auto · The Chinese University of Hong Kong, Shenzhen
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
May 17, 2026Mohammad Mozaffari, Younes Hourri, Mohammad Rastegari +1LLM Pruning Neural Network Pruning
Elastix AI · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
May 15, 2026Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1LLM Pruning LLM Compression
University of Southern California, USA · Inha University, Republic of Korea
Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning
May 14, 2026Jingjing Wang, Xiwen Chen, Wenhui Zhu +6LLM Pruning AI Coding Agents
Clemson University · Morgan Stanley · Arizona State University +1
TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability
May 14, 2026Krish Sharma, Omar Naim, Soumadeep Saha +3Representation Geometry LLM Pruning
ANITI, France · Meta · Apple, USA
Spectral structural distortion reveals redundant neurons in neural networks
May 14, 2026Yongyu WangLLM Pruning Neural Network Compression
Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity
May 13, 2026Cristian Hinostroza, Rodrigo Toro Icarte, Christ Devia +4LLM Pruning LLM Interpretability
1Pontificia Universidad Cat´olica de Chile · 2National Center for Artificial Intelligence (CENIA) · 3Universidad de Chile
STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes
May 13, 2026Chenjun Xu, Zhennan Zhou, Zhan Su +3LLM Pruning LLM Fine-Tuning
University of Washington · University of Montreal
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs
May 12, 2026Jingzhou Jiang, Yi Yang, Kar Yan TamText Embeddings LLM Pruning
The Hong Kong University of Science and Technology
Compute Where it Counts: Self Optimizing Language Models
May 11, 2026Yash Akhauri, Mohamed S. AbdelfattahLLM Inference Efficiency RL for Language Models
Cornell University
A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models
May 10, 2026Djamel BouchaffraLLM Pruning Attention Head Analysis
DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France
Relative Kinetic Utility: Calibrating Cross-Layer Credit for Global Structured LLM Pruning
May 9, 2026Tianhao Qian, Guilin Qi, Jiayu ChenLLM Pruning Structured Sparsity
School of Mathematics, Southeast University Nanjing 210096, China
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Pruning KV Caching
Shanghai Jiao Tong University
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions
May 8, 2026Boyu Shi, Chang Liu, ChuanBao Gao +2LLM Pruning LLM Interpretability
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask
May 7, 2026Liu Hanzuo, Chaofan Lin, Weixuan Sun +4LLM Pruning Sparse Recovery
†Tsinghua University · ‡Tencent
Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models
May 2, 2026Daniel Agyei Asante, Ernie Chang, Yang LiLLM Pruning LLM Compression
Iowa State University · Meta
Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Pruning On-Device Language Model Inference
Meta
Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models
Apr 29, 2026M. K. Khalidi Siam, Md. Tausif-Ul-Islam, Md. Reshad Romim Khan +5LLM Pruning Language Modeling
1BRAC University
Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
Apr 28, 2026Ocean Monjur, Shahriar Kabir Nahin, Anshuman ChhabraLLM Pruning Test-Time Scaling
Bellini College of AI, Cybersecurity, and Computing University of South Florida Tampa, FL, USA
Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
Apr 27, 2026Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3LLM Pruning LLM Inference Acceleration
Neural Superintelligence Lab, MODULABS, Republic of Korea · University of Southern California, United States · Boston University, United States +2
DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
Apr 27, 2026Zahra Dehghanighobadi, Asja FischerLLM Pruning KV Caching
1Ruhr University Bochum · 2UAR Research Center for Trustworthy Data Science and Security
Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Apr 27, 2026Amogh Sheth, Biruk Assefa, Yi Wen Huang +2LLM Pruning Self-Attention
Edison Academy Magnet School · State University of New York College at Plattsburgh · The University of Texas at Austin +2
Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layers
Apr 26, 2026Audrey Cherilyn, Houman SafaaiTransformer Interpretability LLM Pruning
Kempner Institute at Harvard University
SimDiff: Depth Pruning via Similarity and Difference
Apr 21, 2026Yuli Chen, Shuhao Zhang, Fanshen Meng +4LLM Pruning LLM Inference Acceleration
GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models
Apr 21, 2026Ziyang Wang, Jiangfeng Xiao, Chuan Xiao +3LLM Pruning LLM Compression
Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
Apr 19, 2026Yangsong Lan, Hongliang Dai, Piji LiLLM Pruning LLM Compression
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China.
Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution
Apr 18, 2026Qinhao Chen, Linyang He, Nima MesgaraniLLM Pruning Feature Attribution
Columbia University
Optimizing Korean-Centric LLMs via Token Pruning
Apr 17, 2026Hoyeol Kim, Hyeonwoo KimMultilingual Language Model Evaluation LLM Pruning
College of Computing · Independent Researcher
Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs
Apr 17, 2026Wai Man Si, Mingjie Li, Michael Backes +1LLM Pruning LLM Safety Alignment
CISPA Helmholtz Center for Information Security
MOONSHOT : A Framework for Multi-Objective Pruning of Vision and Large Language Models
Apr 14, 2026Gabriel Afriat, Xiang Meng, Shibal Ibrahim +2LLM Pruning Neural Network Pruning
Operations Research Center Massachusetts Institute of Technology · Google · OpenAI +1
Measuring the Redundancy of Decoder Layers in SpeechLLMs
Mar 5, 2026Adel Moumen, Guangzhi Sun, Philip C WoodlandLLM Pruning Transformer
Department of Engineering, University of Cambridge, UK
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization
Mar 1, 2026Theophilus Amaefuna, Hitesh Vaidya, Anshuman Chhabra +1LLM Pruning LLM Compression
1Bellini College of Artificial Intelligence, Cybersecurity and Computing University of South Florida, Tampa, Florida, USA
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
Data-Free Pruning of Self-Attention Layers in LLMs
Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Pruning LLM Compression
School of Computer Science University of St Andrews United Kingdom
PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
Sep 27, 2025Younes Hourri, Mohammad Mozaffari, Maryam Mehri DehnaviLLM Pruning LLM Compression
Department of Computer Science University of Toronto
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1
Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning
May 13, 2025Ren ZhuangGradient-Based Attribution LLM Pruning
School of Information Science and Technology, Hangzhou Normal University
DarwinLM: Evolutionary Structured Pruning of Large Language Models
Feb 11, 2025Shengkun Tang, Oliver Sieberling, Eldar Kurtic +2LLM Pruning Structured Pruning
MBZUAI · ETH Zurich · ISTA +1
LLM Compression by Block Removal with Constrained Binary Optimization
Date pendingDavid Jansen, Roman Rausch, Ali Hashemi +2LLM Pruning LLM Compression
Multiverse Computing · Donostia International Physics Center Ikerbasque Foundation for Science

Jun 2, 2026, cs.LGJ/K move · Enter open · S save

Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning

Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang+4

Shanghai Jiao Tong University

LLM Pruning Model Calibration

Open paper AskPDF arXiv

Post-training pruning compresses large language models to high sparsity using a small unlabelled calibration set, and recent work has concluded that the choice of calibration source has only modest impact on averaged post-pruning accuracy. We ask whether this conclusion survives once calibration impact is evaluated separately across distinct capability dimensions rather than aggregated. Decomposing post-pruning capability into General, Commonsense, Code, and Math, and analysing

n{=}15

calibration sources via Spearman correlations between OIT information metrics and per-dimension retention, we uncover an opposite-sign trade-off: calibration perplexity correlates positively with General retention (

ρ{=}{+}0.71

) but negatively with Math and Code retention (

ρ{=}{-}0.53,\,{-}0.59

;

p{<}0.05

), so no single source can preserve all capabilities. We respond with multi-source calibration mixing, and propose IGSP, an information-guided self-calibration protocol that automates multi-source construction without capability-aligned corpora by minimising 4-gram aggregation and balancing perplexity across dimensions. On LLaMA-3.1-8B at SparseGPT 60% sparsity, a uniform multi-source mix reaches 58.8% total retention, outperforming the best single source (MetaMath, 50.0%) by

+8.8

and the C4 default (40.0%) by

+18.8

; IGSP improves over Self-Cal by

+2.4

and SGS by

+4.8

Momentum

Related topics

Latest papers 113