LLM Compression

LLM: Large Language Model

LLM compression studies how to make language models smaller or cheaper to run while retaining useful abilities. Common approaches include low-bit quantization, pruning weights, layers, or experts, low-rank factorization, distillation into smaller models, and compressing prompts or KV caches. Methods increasingly use calibration data, sensitivity estimates, or adaptive budget allocation, often paired with specialized inference kernels to turn compact representations into practical speedups. A central challenge is that quality, reasoning, reliability, and actual runtime can change in different ways: fewer parameters or tokens do not always mean faster execution or preserved behavior.

Latest papers

283 papers44 papers in the last four weeks, up 52% on the four weeks before. 0.3% of all new papers.

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
Mar 18, 2026Ruibo Fan, Xiangrui Yu, Xinglin Pan +5Model Compression LLM Compression
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR
Capability-Guided Compression: Toward Interpretability-Aware Budget Allocation for Large Language Models
Mar 17, 2026Rishaank GuptaPhase Transitions in Neural Networks Model Compression
Independent Researcher
Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization
Mar 17, 2026Francesco Pio Monaco, Elia Cunegatti, Flavio Vella +1LLM Compression LLM Pruning
University of Trento
Effective Distillation to Hybrid xLSTM Architectures
Mar 16, 2026Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7LLM Compression Language Model Distillation
1NXAI · 2Johannes Kepler University Linz
Knowledge Distillation for Large Language Models
Mar 14, 2026Alejandro Paredes La Torre, Barbara Flores, Diego RodriguezLanguage Model Distillation LLM Compression
Duke University
LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
Mar 13, 2026Jiawei Hao, Zhiwei Hao, Jianyuan Guo +4Mixture-of-Experts Language Models LLM Compression
School of Information and Electronics, Beijing Institute of Technology, Beijing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +2
GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
Mar 12, 2026Xiaoyun Liu, Divya Saxena, Jiannong Cao +3LLM Pruning Structured Pruning
Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India
Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language
Mar 12, 2026Remigiusz Kinas, Paweł Kiszczak, Sergio P. Perez +4LLM Compression Language Modeling
Bielik.AI, Ingenix.ai · Bielik.AI, Vstorm · NVIDIA +3
SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Mar 12, 2026Xinhao Huang, You-Liang Huang, Zeyi WenLLM Compression Low-Rank Compression
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Leech Lattice Vector Quantization for Efficient LLM Compression
Mar 11, 2026Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough +1LLM Quantization LLM Compression
Qualcomm AI Research
Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck
Mar 9, 2026Fabio Valerio Massoli, Andrey Kuzmin, Arash BehboodiEfficient Language Model Reasoning CoT Compression
Qualcomm AI Research
High-Fidelity Pruning for Large Language Models
Mar 9, 2026Yijun Zhu, Jianxin Wang, Chengchao ShenLLM Pruning Neural Network Pruning
Central South University
SCA: Segment-Wise CoT Compression with Answer Alignment
Mar 8, 2026Ye Tian, Hongyu LinCoT Compression Efficient Language Model Reasoning
A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
Mar 8, 2026Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal +3Diffusion Language Models Representation Geometry in Language Models
Qualcomm AI Research, Qualcomm Technologies Inc.
ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning
Mar 6, 2026Mingluo Su, Huan WangLLM Pruning Neural Network Pruning
Westlake University
Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs
Mar 5, 2026Jihao Xin, Tian Lyu, Qilong Pan +2Model Compression LLM Compression
KAUST · HUMAIN
NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
Mar 4, 2026Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7LLM Input Optimization LLM Compression
Pluralis Research
Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models
Mar 3, 2026Shubhangi Upasani, Ravi Shanker Raju, Bo Li +5Prompt Compression Long-Context Language Model Inference
SambaNova AI
3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
Mar 2, 2026Mehdi Makni, Xiang Meng, Rahul MazumderLLM Compression Low-Rank Compression
Operations Research Center Massachusetts Institute of Technology
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization
Mar 1, 2026Theophilus Amaefuna, Hitesh Vaidya, Anshuman Chhabra +1LLM Pruning LLM Compression
Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, Florida, USA
PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training
Feb 26, 2026Yanyi Li, Yimu Zhang, Cong FangLLM Compression Memory-Efficient Training
School of Intelligence Science and Technology, Peking University, China · Institute for Artificial Intelligence, Peking University, China
Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory
Feb 25, 2026Davide EttoriLLM Compression Representation Geometry in Language Models
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
Feb 25, 2026Minhao Jiang, Zhikai Li, Xuewen Liu +3LLM Pruning LLM Compression
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Feb 17, 2026Ziyu Zhao, Tong Zhu, Zhi Zhang +6Mixture-of-Experts Language Models Mixture-of-Experts Pruning
Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory +3
Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
Feb 16, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Memory-Efficient Attention KV-Cache Compression
Sapient Intelligence · Department of Computing Science, University of Alberta
Improving the Performance of Compressed Transformers with Post-Hoc Embedding Alignment
Feb 16, 2026Rohit Raj Rai, Abhishek Dhaka, Amit AwekarAdapter Tuning Representation Alignment
Indian Institute of Technology Guwahati, Guwahati, Assam, India · B.K. Birla Institute of Engineering & Technology, Pilani, Rajasthan, India
QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs
Feb 11, 2026Kanghyun Noh, Jinheon Choi, Yulhwa KimLLM Quantization LLM Compression
Department of Semiconductor Convergence Engineering, Sungkyunkwan University · Department of Electrical and Computer Engineering, Sungkyunkwan University · Department of Semiconductor Systems Engineering, Sungkyunkwan University
LLM Compression by Block Removal with Constrained Binary Optimization
Jan 29, 2026David Jansen, Roman Rausch, Ali Hashemi +2LLM Pruning LLM Compression
Multiverse Computing · Donostia International Physics Center Ikerbasque Foundation for Science
ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits
Jan 20, 2026Aryan KarmoreTernary Quantization Mixture-of-Experts Language Models
Indian Institute of Information Technology, Nagpur

Mar 18, 2026J/K move · Enter open · S save

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

Ruibo Fan, Xiangrui Yu, Xinglin Pan, Zeyu Li+4

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR

Model Compression LLM Compression Memory-Efficient Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Lossless model compression holds tremendous promise for alleviating the memory and bandwidth bottlenecks in bit-exact Large Language Model (LLM) serving. However, existing approaches often result in substantial inference slowdowns due to fundamental design mismatches with GPU architectures: at the kernel level, variable-length bitstreams produced by traditional entropy codecs break SIMT parallelism; at the system level, decoupled pipelines lead to redundant memory traffic. We present ZipServ, a lossless compression framework co-designed for efficient LLM inference. ZipServ introduces Tensor-Core-Aware Triple Bitmap Encoding (TCA-TBE), a novel fixed-length format that enables constant-time, parallel decoding, together with a fused decompression-GEMM (ZipGEMM) kernel that decompresses weights on-the-fly directly into Tensor Core registers. This "load-compressed, compute-decompressed" design eliminates intermediate buffers and maximizes compute intensity. Experiments show that ZipServ reduces the model size by up to 30%, achieves up to 2.21x kernel-level speedup over NVIDIA's cuBLAS, and expedites end-to-end inference by an average of 1.22x over vLLM. ZipServ is the first lossless compression system that provides both storage savings and substantial acceleration for LLM inference on GPUs.