LLM Quantization

LLM: Large Language Model

LLM quantization studies how to represent model weights, activations, and attention key-value caches with fewer bits while preserving useful behavior and making deployment more efficient. Post-training methods such as GPTQ and AWQ use calibration data to round weights, while quantization-aware training, mixed-precision allocation, rotations, and outlier handling aim to reduce errors that accumulate across layers. Work also co-designs number formats and GPU, CPU, and edge-device kernels, evaluating trade-offs with language-model perplexity, zero-shot and reasoning benchmarks, memory, latency, and energy. Four-bit inference is often practical, but performance at very low bit widths remains model- and task-dependent, and an open challenge is ensuring that gains in aggregate accuracy do not hide losses in reasoning, safety, fairness, or uncertainty.

Latest papers

388 papers84 papers in the last four weeks, up 121% on the four weeks before. 0.5% of all new papers.

Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
May 22, 2026Yoshihiko Fujisawa, Yuma Ichikawa, Yudai Fujimoto +2LLM Quantization Memory-Efficient Fine-Tuning
Fujitsu Limited · Institute of Science Tokyo · RIKEN Center for AIP +1
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor
May 19, 2026Xiaocan Li, Shiliang Wu, Zheng ShenLLM Quantization 4-Bit Quantization
Huawei Canada
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
May 19, 2026Haiquan Lu, Zigeng Chen, Gongfan Fang +2LLM Quantization Long-Context Language Model Inference
National University of Singapore
Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
May 19, 2026Jinghe Zhang, Daliang Xu, Chenghua Wang +5Mixed-Precision Quantization LLM Quantization
K-Quantization and its Impact on Output Performance
May 19, 2026Robin Baki Davidsson, Pierre NuguesLLM Quantization LLM Evaluation
Lund University · Lund, Sweden
TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization
May 19, 2026Zukang Xu, Xing Hu, Dawei YangLLM Quantization Rotation-Based Quantization
1Houmo AI.
A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization
May 18, 2026Lei DongLLM Quantization Ternary Quantization
Independent Researcher
GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets
May 18, 2026Zhangyang Yao, Haiyan Zhao, Haoyu Wang +3Mixed-Precision Quantization LLM Quantization
Beihang University · Tsinghua University · ByteDance Inc
MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization
May 18, 2026Le Su, Xing Luo, Zhi JinLLM Quantization Post-Training Quantization
School of Intelligent Systems Engineering Shenzhen Campus of Sun Yat-sen University Shenzhen, Guangdong 518107, China · Department of Frontier Research Peng Cheng Laboratory Shenzhen, Guangdong 518055, China
Attention Sinks and Outliers in Attention Residuals
May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM Quantization Attention Residual Connections
Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5
WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
May 17, 2026Dongyue Li, Zechun Liu, Kai Yi +6LLM Quantization Quantization-Aware Training
1Northeastern University, MA · 2Meta AI, CA.
Measuring Maximum Activations in Open Large Language Models
May 15, 2026Luxuan Chen, Han Tian, Xinran Chen +9LLM Quantization Activation Outliers
1Shanghai Jiao Tong University · 2Baidu Inc. · 4Nankai University +1
Widening the Gap: Exploiting LLM Quantization via Outlier Injection
May 14, 2026Xiaohua Zhan, Kazuki Egashira, Robin Staab +2LLM Quantization LLM Backdoor Attacks
ETH Zurich
A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models
May 14, 2026Earl KillianMixed-Precision Quantization LLM Quantization
XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
May 14, 2026Thomas WittLLM Quantization Vector Quantization
Gemini Stiftung Leipzig
High-Rate Quantized Matrix Multiplication II
May 13, 2026Or Ordentlich, Yury PolyanskiyLLM Quantization Rate-Distortion Theory
the Hebrew University of Jerusalem, Israel · MIT, USA
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
May 13, 2026Lingchao Zheng, Yuwei Fan, Jun Li +5LLM Quantization LLM Inference Acceleration
Huawei
Grid Games: The Power of Multiple Grids for Quantizing Large Language Models
May 12, 2026Vage Egiazarian, Erik Schultheis, Andrei Panferov +3LLM Quantization 4-Bit Quantization
ISTA · Aril Computer · ETH Zürich +1
SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
May 12, 2026Chengzhu Bao, Xianglong Yan, Zhiteng Li +3LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent Hunyuan
PRISM: A Geometric Risk Bound for Decomposing Drift into Scale, Shape, and Head
May 12, 2026Chieh-Yen Lin, Shao-Hua SunLLM Quantization LLM Evaluation
Appier AI Research · National Taiwan University
Efficient LLM-based Advertising via Model Compression and Parallel Verification
May 12, 2026Wenxin Dong, Chang Gao, Guanghui Yu +9LLM Quantization LLM Compression
Baidu Inc. Beijing, China
ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
May 11, 2026Ryan Lucas, Mehdi Makni, Xiang Meng +2LLM Quantization Alternating Direction Method of Multipliers
MIT Operations Research Center · MIT Sloan School of Management · MIT Center for Statistics
Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
May 11, 2026Yao Shu, Zilin ZhuLLM Quantization Zeroth-Order Optimization
HKUST-GZ
BCJR-QAT: A Differentiable Relaxation of Trellis-Coded Weight Quantization
May 11, 2026Venugopalan IyengarLLM Quantization Quantization-Aware Training
Theory-optimal Quantization Based on Flatness
May 11, 2026Xiusheng Huang, Zhe Li, Xuanwu Yin +5LLM Quantization Activation Quantization
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2
Pretraining large language models with MXFP4 on Native FP4 Hardware
May 11, 2026Musa Cim, Poovaiah Palangappa, Miro Hodak +3Language Model Pretraining LLM Quantization
1The Pennsylvania State University · 2Advanced Micro Devices, Inc.
Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs
May 9, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +2LLM Quantization Post-Training Quantization
1Harbin Institute of Technology, Harbin, China · 2Tsinghua University, Beijing, China
LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss
May 9, 2026Euntae Choi, Sumin Song, Sungjoo YooLLM Quantization Quantization-Aware Training
Seoul National University
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
May 9, 2026Beshr IslamBouli, David JinLLM Quantization 4-Bit Quantization
Massachusetts Institute of Technology, Cambridge, MA, USA.

May 22, 2026J/K move · Enter open · S save

Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning

Yoshihiko Fujisawa, Yuma Ichikawa, Yudai Fujimoto, Akira Sakai+1

Fujitsu Limited · Institute of Science Tokyo · RIKEN Center for AIP+1

LLM Quantization Memory-Efficient Fine-Tuning LLM Fine-Tuning Adapter Tuning Low-Rank Adaptation Quantization-Aware LoRA+1

Open paper AskPDF arXiv

On-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task-specific LoRA adapter. In the unmerged adapter-mode setting, however, the adapter is more than a compact storage module; it introduces an additional dense floating-point branch, maintains a trainable state for local updates, and acts as a unit of communication and hot-swapping.We introduce LoRDBA, a LoRA-compatible adapter that replaces both low-rank factors with binary sign carriers while representing magnitudes through lightweight, channel-wise scales, converting the dense adapter branch into two sign-accumulation matrix multiplications interleaved with channel-wise scaling. A finite-sample analysis shows that reconstruction quality is governed by the residual-to-magnitude ratio of the original LoRA factors. In adapter-mode experiments, LoRDBA outperforms low-bit baselines at matched model sizes while matching fp16 LoRA quality in selected regimes. The unmerged adapter incurs at most 8% prefill latency overhead at matched rank r=16 despite an over 10x reduction in adapter footprint, with moderate training memory overhead of approximately 1.6x that of fp16 LoRA.