LLM Compression

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 92% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 178

All topics

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
Jul 8, 2026Yazdan Jamshidi, Alexey ShvetsLLM Pruning LLM Compression
Palo Alto Networks
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
Jul 5, 2026Akhiad Bercovich, Talor Abramovich, Daniel Afrimi +67Mixture-of-Experts Pruning LLM Pruning
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3
Parameter Golf: What Really Works?
Jul 1, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLLM Evaluation LLM Compression
Geometric Intelligence Research Lab., Department of Electrical Engineering and Computer Science University of Wyoming
GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache
Jul 1, 2026Soosung Kim, Minjae Park, Eui-Young Chung +1LLM Compression Residual VQ
Department of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea · Department of Systems Semiconductor Engineering, Yonsei University, Seoul, Republic of Korea
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
Jun 30, 2026Dohyeon Kwon, Youngjin ParkMixture-of-Experts Language Models LLM Compression
AXOps Team, TeamSparta Inc., Seoul, South Korea
When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis
Jun 28, 2026Hoyoung Lee, Suhwan Park, Seunghan Lee +15LLM Compression LLM Auditing
UNIST · LinqAlpha · LG AI Research +9
DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training
Jun 27, 2026Dong Wang, Wenwu Tang, Yun Cheng +1Language Model Pretraining LLM Compression
Graz University of Technology, Austria · Swiss Data Science Center, Switzerland
MultiHashFormer: Hash-based Generative Language Models
Jun 26, 2026Huiyin Xue, Atsuki Yamaguchi, Nikolaos AletrasLLM Compression Decoder-Only Language Models
School of Computer Science, University of Sheffield, United Kingdom
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
Jun 26, 2026Fan Mo, Yuxuan Han, Geng Zhang +2Mixture-of-Experts Pruning LLM Pruning
National University of Singapore
Output-Space Allocation Costs for Calibration-Guided LLM Compression: An Empirical Study
Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Model Compression LLM Compression
Analemma
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
Jun 25, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +2LLM Quantization Ternary Quantization
1Intel Labs China
EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression
Jun 24, 2026Ke Xu, Jiaqi Wan, Wenhao Hu +2LLM Pruning LLM Compression
Scaling Laws for Task-Specific LLM Distillation
Jun 23, 2026Lavinia Ghita, Dhruv Desai, Ioana BoierLLM Pruning LLM Compression
NVIDIA
CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Jun 23, 2026Morayo Danielle Adeyemi, Ryan A. Rossi, Franck DernoncourtLLM Evaluation LLM Compression
Independent · Adobe Research
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Jun 22, 2026Mahmoud Safari, Frank HutterLLM Pruning LLM Compression
University of Freiburg · Prior Labs · ELLIS Institute Tübingen
TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation
Jun 20, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4LLM Compression Language Model Distillation
Hanoi University of Science and Technology · VNU University of Engineering and Technology · University of Oregon +1
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Jun 20, 2026Chao Han, Yongjie Du, Junjie Tan +1LLM Compression Low-Rank Matrix Decomposition
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo
Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
Jun 18, 2026Nico Harder, Daniel Becking, Karsten Mueller +1LLM Compression Low-Rank Matrix Decomposition
Fraunhofer HHI, Berlin, Germany
Dual Dimensionality for Local and Global Attention
Jun 17, 2026Zhiyuan Wang, Xuan Luo, Sirui Zeng +1LLM Compression Decoder-Only Language Models
UC Santa Barbara
Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models
Jun 16, 2026Ramprasath Ganesaraja, Sahil Dilip Panse, Swathika NLLM Quantization Ternary Quantization
EdgeVerve Systems Limited
The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
Jun 16, 2026Rui Wen, Lu Sun, Jiayang Liu +3Open-Ended Generation LLM Evaluation
Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
Jun 15, 2026Martin JaggiMixture-of-Experts Language Models LLM Compression
EPFL
Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective
Jun 13, 2026Tao Jing, Ningxin Wu, Chen Kang +3LLM Pruning LLM Compression
Persona-Pruner: Sculpting Lightweight Models for Role-Playing
Jun 12, 2026Jinsu Kim, Jihoon Tack, Noah Lee +1LLM Pruning Large Language Model-Based Role-Play Simulation
Department of Artificial Intelligence, Korea University, Seoul, South Korea · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models
Jun 10, 2026Changyue Wang, Weihang Su, Qingyao Ai +5LLM Compression Efficient Language Model Inference
Department of Computer Science and Technology, Tsinghua University
Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs
Jun 9, 2026Ruixuan Huang, Jinyuan Shi, Hantao Huang +5Continual Learning for LLMs LLM Compression
1HKUST · 2Moffett AI · 3Independent Contributor
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · University of Maryland · Princeton University +3
EinSort: Sorting is All We Need for Tensorizing LLM
Jun 7, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangTensor Networks LLM Compression
Mitsubishi Electric Research Laboratories (MERL), 20 Broadway, Cambridge, MA 02139, USA.
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Jun 6, 2026Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides +3Text Embeddings LLM Compression
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · AIP, RIKEN, Tokyo, Japan.

Jul 8, 2026, cs.CLJ/K move · Enter open · S save

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

Yazdan Jamshidi, Alexey Shvets

Palo Alto Networks

LLM Pruning LLM Compression Neural Network Pruning

Open paper AskPDF arXiv

One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to

\pm 5\%

around the target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs,

p < 0.001

). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.

Momentum

Related topics

Latest papers 178