LLM Pruning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 116

All topics

Less is MoE: Trimming Experts in Domain-Specialist Language Models
Jun 4, 2026Haoze He, Xinkai Zou, Xuan Jiang +4Mixture-of-Experts Pruning LLM Pruning
Carnegie Mellon University · UCSD · MIT
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
Jun 3, 2026Yuhao ZhangMixture-of-Experts Pruning LLM Pruning
Beihang University
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
Jun 3, 2026Jiangyang He, Shaolin Zhu, Deyi XiongMixture-of-Experts Pruning LLM Pruning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China
Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning
Jun 2, 2026Hu Xu, Zhaolong Xing, Congcong Liu +5LLM Pruning Model Calibration
Shanghai Jiao Tong University
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
Jun 1, 2026Elia Cunegatti, Marcus Vukojevic, Erik Nielsen +1LLM Pruning LLM Compression
University of Trento
Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search
Jun 1, 2026Cheonjun ParkLLM Pruning LLM Compression
Hankuk University of Foreign Studies
Before Parc Fermé: RL-Time Pruning for Efficient Embodied LLMs in Autonomous Driving
May 29, 2026Luca Benfenati, Ali Azimi, Matteo Risso +3LLM Pruning Language Model-Based Control
Department of Control and Computer Engineering, Politecnico di Torino · Department of Mechanical and Aerospace Engineering, Politecnico di Torino
AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
May 28, 2026Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan KandemirLLM Pruning Vision-Language Models
The Pennsylvania State University
Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
May 28, 2026Junlin He, Yihong Tang, Tong Nie +5LLM Pruning LLM Compression
The Hong Kong Polytechnic University, Hong Kong SAR, China · McGill University, Montreal, QC, Canada
Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules
May 27, 2026Karim Galliamov, Rochelle Choenni, Ivan TitovLLM Pruning LLM Compression
University of Amsterdam · University of Edinburgh
PrunePath: Towards Highly Structured Sparse Language Models
May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts Pruning LLM Pruning
Department of Applied Mathematics, The Hong Kong Polytechnic University
Pruning and Distilling Mixture-of-Experts into Dense Language Models
May 27, 2026Junhyuck Kim, Jihun Yun, Haechan Kim +3Mixture-of-Experts Pruning LLM Pruning
1KRAFTON · 2KAIST
Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
May 27, 2026Liu O. Martin, Lucas Bandarkar, Nanyun PengMixture-of-Experts Pruning LLM Pruning
University of California, Los Angeles
Locality-Aware Redundancy Pruning for LLM Depth Compression
May 27, 2026Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3LLM Pruning LLM Compression
University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
May 26, 2026Fengfa Li, Hongjin Ji, Yifeng Ding +2Mixture-of-Experts Pruning LLM Pruning
Li Auto · The Chinese University of Hong Kong, Shenzhen
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
May 17, 2026Mohammad Mozaffari, Younes Hourri, Mohammad Rastegari +1LLM Pruning Neural Network Pruning
Elastix AI · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
May 15, 2026Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1LLM Pruning LLM Compression
University of Southern California, USA · Inha University, Republic of Korea
Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning
May 14, 2026Jingjing Wang, Xiwen Chen, Wenhui Zhu +6LLM Pruning AI Coding Agents
Clemson University · Morgan Stanley · Arizona State University +1
TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability
May 14, 2026Krish Sharma, Omar Naim, Soumadeep Saha +3Representation Geometry LLM Pruning
ANITI, France · Meta · Apple, USA
Spectral structural distortion reveals redundant neurons in neural networks
May 14, 2026Yongyu WangLLM Pruning Neural Network Compression
Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity
May 13, 2026Cristian Hinostroza, Rodrigo Toro Icarte, Christ Devia +4LLM Pruning LLM Interpretability
1Pontificia Universidad Cat´olica de Chile · 2National Center for Artificial Intelligence (CENIA) · 3Universidad de Chile
STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes
May 13, 2026Chenjun Xu, Zhennan Zhou, Zhan Su +3LLM Pruning LLM Fine-Tuning
University of Washington · University of Montreal
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs
May 12, 2026Jingzhou Jiang, Yi Yang, Kar Yan TamText Embeddings LLM Pruning
The Hong Kong University of Science and Technology
Compute Where it Counts: Self Optimizing Language Models
May 11, 2026Yash Akhauri, Mohamed S. AbdelfattahLLM Inference Efficiency RL for Language Models
Cornell University
A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models
May 10, 2026Djamel BouchaffraLLM Pruning Attention Head Analysis
DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France
Relative Kinetic Utility: Calibrating Cross-Layer Credit for Global Structured LLM Pruning
May 9, 2026Tianhao Qian, Guilin Qi, Jiayu ChenLLM Pruning Structured Sparsity
School of Mathematics, Southeast University Nanjing 210096, China
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Pruning KV Caching
Shanghai Jiao Tong University
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST

Jun 4, 2026, cs.LGJ/K move · Enter open · S save

Less is MoE: Trimming Experts in Domain-Specialist Language Models

Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding+3

Carnegie Mellon University · UCSD · MIT

Mixture-of-Experts Pruning LLM Pruning LLM Inference Acceleration Structured Pruning Mixture-of-Experts Models Neural Network Pruning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning. We trace this failure to the granularity of compression: important capabilities are distributed across experts but concentrated in FFN sparse intermediate dimensions. To identify these dimensions, we use Fisher importance which outperforms activation-, router-score-, and magnitude-based alternatives, and identifies tiny sets of task-critical dimensions: in Qwen1.5-MoE, removing as few as 12 of 1.35M routed-FFN intermediate dimensions collapses GSM8K accuracy while largely preserving factual-knowledge performance. Building on this, we propose Fisher-MoE, which operates within FFN to remove intermediate dimensions ranked by Fisher importance. At the same 50% MoE compression ratio, Fisher-MoE preserves model capability, while reducing weight memory by ~45% and improving inference throughput by 21%. These findings suggest intermediate dimension granularity is an effective unit for both compression and ranking where capability concentrates in MoE models.

Momentum

Related topics

Latest papers 116