LLM Quantization

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

33 papers in the last four weeks, up 43% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 251

All topics

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models
Jul 16, 2026Ryona NodaLLM Quantization Post-Training Quantization
Ryo Noda
PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
Jul 16, 2026Hyunwoo Oh, Suyeon Jang, Hanning Chen +4Mixed-Precision Quantization LLM Quantization
University of California, Irvine
Quantize with Confidence? An Empirical Study of Quantization for Code Generation
Jul 15, 2026Saima Afrin, Md. Zahidul Haque, Antonio MastropaoloLLM Quantization Code Generation Evaluation
AURA Lab · Department of Computer Science · William & Mary +1
ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level
Jul 15, 2026Chethan Reddy G. PLLM Quantization Ternary Quantization
Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate
Jul 14, 2026Joshua HillMixed-Precision Quantization LLM Quantization
Baseten Labs, Inc. · David R. Cheriton School of Computer Science, University of Waterloo
Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
Jul 13, 2026Weijia Han, Lisha QuLLM Quantization GPU Kernel Optimization
University of Washington
Reliability Scaling Laws for Quantized Large Language Models
Jul 12, 2026Sirine Ayadi, Sándor Daróczi, Stephan Günnemann +1LLM Quantization Language Model Calibration
School of Computation, Information and Technology, Technical University of Munich Munich Data Science Institute · Pruna AI
Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
Jul 10, 2026Renuka Oladri, Mohan Vamsi Varadaraju Priya, Jerry WuLLM Quantization LLM Evaluation
Science Academy University of Maryland College Park, MD, USA · Electrical and Computer Engineering University of Maryland College Park, MD, USA
KronQ: LLM Quantization via Kronecker-Factored Hessian
Jul 8, 2026Donghyun Lee, Yuhang Li, Ruokai Yin +1Mixed-Precision Quantization LLM Quantization
University of Southern California · Yale University
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Language Model Pretraining Mixed-Precision Quantization
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3
HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
Jul 5, 2026Hui Dong, Yanzhao Li, Jie Gao +5Softmax Attention LLM Quantization
Huawei Technologies
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Jul 5, 2026Jaeyeon Kim, Jewon Lee, Bo-Kyeong KimLLM Quantization Speculative Decoding
Nota Inc.
ELiTeFormer: An Efficient Transformer for FPGAs
Jul 4, 2026Victor Agostinelli, Nicolas Bohm Agostini, Antonino TumeoLLM Quantization Ternary Quantization
Oregon State University Corvallis, Oregon, USA · Pacific Northwest National Laboratory Richland, Washington, USA
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
Jul 3, 2026Hamish OgilvyMixed-Precision Quantization LLM Quantization
Algolia Research
$\text{Log}_\text{b}$ Quant: Quantizing Language Models in Logarithmic Space
Jul 1, 2026Jeremias Bohn, Tizian Dippold, Mahdi Koubaa +2LLM Quantization 4-Bit Quantization
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization
Jul 1, 2026Fei Wang, Chao Xue, Taoran Liu +3Mixed-Precision Quantization LLM Quantization
South China University of Technology · Ocean University of China · Shenzhen Campus of Sun Yat-sen University +1
The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis
Jun 28, 2026Hari Prasad, Ritam PalLLM Safety Benchmarks LLM Quantization
Conscious Engines
FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks
Jun 27, 2026Aoying Zheng, Anqi Du, Zizhuang Deng +1LLM Quantization LLM Backdoor Attacks
School of Cyber Science and Technology Shandong University · Suzhou Research Institute of Shandong University
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
Jun 25, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +2LLM Quantization Ternary Quantization
1Intel Labs China
BitNet Text Embeddings
Jun 24, 2026Zhen Li, Xin Huang, Liang Wang +8LLM Quantization Embedding Compression
Peking University · Microsoft Research
Agentic evolution of physically constrained foundation models
Jun 24, 2026Jiangwei Zhang, Wen Sun, Chong Wang +7Model Compression LLM Quantization
State Key Laboratory of Cyberspace Security Defense, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, 100084, China. · University of Chinese Academy of Sciences, Beijing, 101408, China. · Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China.
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
Jun 24, 2026Xinyu Lian, Walid Krichene, Beichen Huang +4LLM Inference Efficiency LLM Quantization
University of Illinois Urbana-Champaign · Microsoft · Anyscale +1
GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation
Jun 22, 2026Jette Oberländer, Jan Finkbeiner, Catherine M. Schöfmann +1Mixed-Precision Quantization LLM Quantization
Fakultät für Informatik, RWTH Aachen, Aachen, 52074, Germany · Peter Grünberg Institut, Forschungszentrum Jülich GmbH, Jülich, 52425, Germany · Fakultät für Elektrotechnik und Informationstechnik, RWTH Aachen, Aachen, 52062, Germany
HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
Jun 22, 2026Yuval Domb, Hadar Sackstein, Tomer SolbergLLM Quantization Diffusion Model Quantization
On the Expressive Power of Weight Quantization in Large Language Models
Jun 20, 2026Shao-Qun ZhangLLM Quantization Weight-Only Quantization
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China. · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China.
Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach
Jun 19, 2026Pedro M. R. Pereira, Felipe A. P. de Figueiredo, Rausley A. A. de SouzaLLM Quantization Rotation-Based Quantization
National Institute of Telecommunications (Inatel), Santa Rita do Sapucaí, MG 37536-001, Brazil
An Empirical Study of OpenPangu Quantization on Ascend NPUs
Jun 19, 2026Tong Shi, Jiacheng Wang, Hui Xie +4LLM Quantization AI Accelerator Inference
State Key Laboratory of Complex & Critical Software Environment, Beihang University · School of Artificial Intelligence, Beihang University · School of Computer Science and Engineering, Beihang University
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Jun 18, 2026Qian Zhao, Kunlong Chen, Changxin Tian +9Language Model Pretraining LLM Quantization
Ling Team, Ant Group
Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
Jun 17, 2026Miloš Nikolić, Ali Hadi Zadeh, Enrique Torres Sanchez +1LLM Quantization LLM Evaluation
ByteShape · University of Toronto · Vector Institute for Artificial Intelligence
Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models
Jun 16, 2026Ramprasath Ganesaraja, Sahil Dilip Panse, Swathika NLLM Quantization Ternary Quantization
EdgeVerve Systems Limited

Jul 16, 2026, cs.NEJ/K move · Enter open · S save

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models

Ryona Noda

Ryo Noda

LLM Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Layer-wise post-training quantization of large language models minimizes each layer's reconstruction error in isolation, allowing quantization errors to accumulate across depth and causing severe degradation in extreme low-bit regimes. We formulate quantization as a joint optimization over the discrete codes and scales of all layers, driven by two mechanisms: (i) cross-layer error compensation, which maintains the network-level accumulated error through the recursion e_{l+1} = A_l e_l + q_l, with a propagation operator A_l derived from the layer's input differential and a local quantization residual q_l evaluated at teacher features; and (ii) finite-sample feature-statistics matching, which aligns means, projected covariances, and centered empirical kernels between the full-precision and quantized networks under relative normalization. We prove that instantiating the propagation operator as a finite difference of the quantized network makes the recursion exact for arbitrary nonlinear layers, enabling an efficient forward-difference implementation. Binary weights are optimized via a mirror-descent parameterization u = tanh(beta*z) with annealed inverse temperature and group-wise log-scales. On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher, outperforming logit distillation (14.09 +/- 0.53; 32 percent relative, more than 8 sigma over 3 seeds) and layer-local reconstruction by two orders of magnitude. The same objective transfers unchanged to 4-bit quantization (1.060 vs. 1.088 for layer-local). Out-of-domain evaluations (C4, CNN/DailyMail) show the advantage of error compensation grows off-domain, while statistics matching keeps feature-statistics discrepancy low off-domain (0.42-0.88 vs. 1.41-2.99 without it), revealing a complementary division of labor between the two mechanisms.

Momentum

Related topics

Latest papers 251