LLM Compression

LLM: Large Language Model

LLM compression studies how to make language models smaller or cheaper to run while retaining useful abilities. Common approaches include low-bit quantization, pruning weights, layers, or experts, low-rank factorization, distillation into smaller models, and compressing prompts or KV caches. Methods increasingly use calibration data, sensitivity estimates, or adaptive budget allocation, often paired with specialized inference kernels to turn compact representations into practical speedups. A central challenge is that quality, reasoning, reliability, and actual runtime can change in different ways: fewer parameters or tokens do not always mean faster execution or preserved behavior.

Latest papers

283 papers44 papers in the last four weeks, up 52% on the four weeks before. 0.3% of all new papers.

ReAD: Reinforcement-Guided Capability Distillation for Large Language Models
May 11, 2026Xueqi Cheng, Xugui Zhou, Tyler Derr +1LLM Compression Language Model Distillation
Florida State University · Louisiana State University · Vanderbilt University
SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
May 11, 2026Anton Plaksin, Sergei Krutikov, Sergei Skvortsov +1LLM Compression Speculative Decoding
Nebius, Amsterdam, Netherlands.
Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
May 10, 2026Jiwei Tang, Zhijing Huang, Xinyu Zhang +5LLM Compression Position Bias
Sun Yat-sen University · Hong Kong Polytechnic University · Beijing Normal–Hong Kong Baptist University
RuPLaR : Efficient Latent Compression of LLM Reasoning Chains with Rule-Based Priors From Multi-Step to One-Step
May 10, 2026Xiaocheng Luo, Kang Wang, Zaifu Zhan +2LLM Compression CoT Compression
School of Computer Science and Technology, Soochow University, Suzhou, 215006, Jiangsu, PR China · Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, 55455, MN, USA
Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression
May 9, 2026Hengyi Zhu, Zhendong Mi, Grace Li Zhang +1LLM Compression Low-Rank Matrix Decomposition
1Stevens Institute of Technology, Hoboken, USA · 2Technical University of Darmstadt, Darmstadt, Germany
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Efficient Transformer Inference LLM Compression
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1
Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
May 8, 2026Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan +17LLM Compression LLM Post-Training
NVIDIA
Layer Collapse in Diffusion Language Models
May 7, 2026Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei LiuLLM Compression Diffusion Language Models
Tübingen AI Center · Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
May 7, 2026Shen Xu, Xiangwen Zhuge, Zhe Xu +3LLM Compression Memory-Efficient Inference
Tsinghua University · Huazhong University of Science and Technology
Telegraph English: Semantic Prompt Compression via Structured Symbolic Rewriting
May 6, 2026Mikhail L. Arbuzov, Sisong Bei, Ziwei Dong +2LLM Compression Prompt Compression
Independent Researcher · Palo Alto Networks
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
May 5, 2026Mohammed Sabry, Anya BelzLLM Compression Low-Rank Adaptation
ADAPT Centre, Dublin City University, Ireland
Complexity Horizons of Compressed Models in Analog Circuit Analysis
May 4, 2026Pacome Simon MbonimpaLLM Evaluation LLM Compression
Carnegie Mellon University-Africa
Mapping Text to Multiplex Graph: Prompt Compression as Lévy Walk-Guided Graph Pruning
May 4, 2026Yaxin Gao, Yao Lu, Jinhong Deng +7LLM Compression Long-Context Modeling
EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer
May 3, 2026Hao Zhang, Zhibin Zhang, Guangxin Wu +3LLM Compression Language Model Distillation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models
May 2, 2026Daniel Agyei Asante, Ernie Chang, Yang LiLLM Pruning LLM Compression
Iowa State University · Meta
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
Model Compression with Exact Budget Constraints via Riemannian Manifolds
May 1, 2026Michael Helcig, Dan AlistarhModel Compression Riemannian Optimization
ETH Zürich · IST Austria
Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
Apr 29, 2026Zhenyu Zhao, Sander Land, Daniel M. Bikel +1LLM Compression LLM Interpretability
Writer, Inc.
Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models
Apr 28, 2026Ajmain Inqiad Alam, Palash Roy, Chanchal K. Roy +2Software Engineering LLM Compression
University of Saskatchewan, Canada
Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2
Apr 28, 2026Chirag ShindeModel Compression LLM Quantization
Independent Researcher
From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors
Apr 25, 2026Yitian Zhou, Chaoning Zhang, Jiaquan Zhang +6LLM Compression Long-Context Language Model Inference
School of Computer Science and Engineering University of Electronic Science and Technology of China · School of Information and Software Engineering, University of Electronic Science and Technology of China · Department of Computer Science and Engineering, Kyung Hee University +3
Hyperloop Transformers
Apr 23, 2026Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon KimLLM Compression Hyper-Connections
Massachusetts Institute of Technology
FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression
Apr 22, 2026Ye Qiao, Yian Wang, Zhiheng Chen +2LLM Compression GPU Acceleration
University of California, Irvine Irvine, California, USA
Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales
Apr 22, 2026Samuel SalfatiLLM Compression Post-Training Quantization
fraQtl AI Research
GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models
Apr 21, 2026Ziyang Wang, Jiangfeng Xiao, Chuan Xiao +3LLM Pruning LLM Compression
Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University
Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations
Apr 20, 2026Yunjia Xi, Menghui Zhu, Jianghao Lin +4LLM Compression Embedding Compression
Shanghai Jiao Tong University Shanghai, China · Huawei Noah’s Ark Lab Shanghai, China · Antai College of Economics and Management, Shanghai Jiao Tong University Shanghai, China
Predicting LLM Compression Degradation from Spectral Statistics
Apr 20, 2026Mingxue XuLLM Compression Low-Rank Matrix Decomposition
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom
Neural Garbage Collection: Learning to Forget while Learning to Reason
Apr 20, 2026Michael Y. Li, Jubayer Ibn Hamid, Emily B. Fox +1LLM Compression Memory-Efficient Optimization
Stanford University
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
Apr 19, 2026Yangsong Lan, Hongliang Dai, Piji LiLLM Pruning LLM Compression
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China.
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
Apr 18, 2026Junlin Li, Shuangyong Song, Guodong Du +6LLM Quantization LLM Compression
Harbin Institute of Technology, Shenzhen, China · TeleAI of China Telecom · The Hong Kong Polytechnic University +1

May 11, 2026J/K move · Enter open · S save

ReAD: Reinforcement-Guided Capability Distillation for Large Language Models

Xueqi Cheng, Xugui Zhou, Tyler Derr, Yushun Dong

Florida State University · Louisiana State University · Vanderbilt University

LLM Compression Language Model Distillation

Open paper AskPDF arXiv

Capability distillation applies knowledge distillation to selected model capabilities, aiming to compress a large language model (LLM) into a smaller one while preserving the abilities needed for a downstream task. However, most existing methods treat capabilities as independent training targets and overlook how improving one capability can reshape the student's broader capability profile, especially when multiple abilities jointly determine task success. We study capability distillation under a fixed token budget and identify two consistent patterns: distillation induces systematic, budget-dependent cross-capability transfer, and additional budget often brings limited task-relevant gains while sometimes degrading other useful abilities. Building on these insights, we propose ReAD, a Reinforcement-guided cApability Distillation framework that explicitly accounts for capability interdependence. ReAD first infers task-essential capabilities, then generates capability-targeted supervision on the fly, and finally uses an uncertainty-aware contextual bandit to adaptively allocate the distillation budget based on expected utility gains. Extensive experiments show that ReAD improves downstream utility under the same token budget while reducing harmful spillover and wasted distillation effort compared to strong baselines. Our code is publicly available at https://github.com/LabRAI/ReAD.