LLM Quantization

LLM: Large Language Model

LLM quantization studies how to represent model weights, activations, and attention key-value caches with fewer bits while preserving useful behavior and making deployment more efficient. Post-training methods such as GPTQ and AWQ use calibration data to round weights, while quantization-aware training, mixed-precision allocation, rotations, and outlier handling aim to reduce errors that accumulate across layers. Work also co-designs number formats and GPU, CPU, and edge-device kernels, evaluating trade-offs with language-model perplexity, zero-shot and reasoning benchmarks, memory, latency, and energy. Four-bit inference is often practical, but performance at very low bit widths remains model- and task-dependent, and an open challenge is ensuring that gains in aggregate accuracy do not hide losses in reasoning, safety, fairness, or uncertainty.

Latest papers

388 papers84 papers in the last four weeks, up 121% on the four weeks before. 0.5% of all new papers.

CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models
Oct 27, 2026Zukang Xu, Zhixiong Zhao, Xing Hu +3LLM Quantization Vector Quantization
Houmo AI · Peking University · Southeast University
Efficient Quantization Training with Stochastic Rounding via Structured Subspace Randomness
Oct 27, 2026Zeyu Wang, Yiheng Duan, Tao Chen +5Stochastic Rounding Low-Precision Training
Google, Mountain View, CA, USA
Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs
Oct 26, 2026Tianqi Chen, Xunchao Li, Linchuan Xie +4Memory-Augmented Language Models LLM Quantization
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent
End-to-End MXFP8 Reinforcement Learning on Blackwell: What Matching Buys, When It Pays Off, and How It Fails
Oct 26, 2026Wengee ZhangRL Fine-Tuning Mixed-Precision Quantization
ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning
Oct 26, 2026Rhea Sanjay Sukthanker, Pashmina CameronLLM Quantization Low-Bit Quantization
Microsoft Applied Sciences Group
Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State
Oct 25, 2026Pierre-Olivier BoninBinary Neural Networks Efficient Language Model Training
Independent Researcher
ResCom: Activation Residual Compensation for Large Language Model Quantization
Oct 25, 2026Ning Yang, Bohan Cui, Zhuowen Liu +1LLM Quantization Quantization Error Compensation
Institute of Automation, Chinese Academy of Sciences · Fudan University · The Chinese University of Hong Kong +1
RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches
Oct 24, 2026Tianqi Chen, Xunchao Li, Linchuan Xie +4LLM Quantization Attention Quantization
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent
BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization
Oct 23, 2026David Jin, Beshr IslamBouli, Tarushii Goel +2LLM Quantization Post-Training Quantization
Massachusetts Institute of Technology
Training an MoE LLM in MXFP4: How We Closed the Gap to BF16
Oct 23, 2026Han Wang, Zhitao Wang, Natalia Frumkin +4LLM Quantization Low-Precision Training
Advanced Micro Devices, Inc.
MPQ: A Message-Passing View of Post-Training Quantization
Oct 23, 2026Yanshuo Chen, Chenxi Liu, Reza Shirkavand +1Quantization LLM Quantization
Department of Computer Science, University of Maryland, College Park, MD 20742
Low-Bit N-Gram Embeddings for Language Models
Oct 22, 2026Stephen Chung, Wenyu DuLLM Compression Low-Bit Quantization
DualverseAI
SQuAT: Spherical Lattice Vector Quantization-Aware Training
Oct 22, 2026Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa +4LLM Quantization Vector Quantization
University of Copenhagen · Qualcomm AI Research
FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization
Oct 21, 2026Faraz Tahmasebi, Haocheng Xu, Andrew Trauger +4LLM Inference Acceleration LLM Quantization
University of California, Irvine, Irvine, CA, USA
Effects of Model Scale and Quantization on Social Bias and Abstention in Question Answering
Oct 21, 2026Kuniko Paxton, Ifeoluwa Wuraola, Dhavalkumar ThakkerSocial Bias in Language Models Language Model Bias Evaluation
University of Hull Cottingham Road, Hull, HU6 7RX, United Kingdom
PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
Oct 21, 2026Patrick Haller, Filipe Laitenberger, Alan AkbikRecurrent-State Quantization LLM Quantization
Humboldt-Universität zu Berlin
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Oct 21, 2026Boyuan Zheng, Jing Xiong, Chenwei Wu +8LLM Quantization Quantization-Aware Training
University of Michigan, Ann Arbor · University of California, Santa Barbara · Meta AI
Compressing Onboard Language Models for Space Robots under a Behavioral-Fidelity Requirement
Oct 21, 2026Jia Deng, Zizhe Wang, Xin Lu +2LLM Compression LLM Quantization
Astroup Lab, Beijing, China
Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference
Oct 20, 2026Xiuyi QinLLM Inference Acceleration Structured Sparsity
Independent Researcher, San Jose, CA, USA.
Beyond 4 bits: Data-Centric Post-training Quantization and Recovery
Oct 20, 2026Rhea Sanjay Sukthanker, Riccardo Grazzi, James Hensman +1LLM Quantization Post-Training Quantization
Applied Sciences Group (ASG), Microsoft · Microsoft Research
ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models
Oct 20, 2026Jiahui Ning, Liang Chen, Yuchen Liang +2LLM Compression LLM Evaluation
City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute +1
E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks
Oct 20, 2026Xinyu Lei, Hong Huang, Jiebin Zhang +6LLM Quantization Quantization-Aware Training
Peking University · City University of Hong Kong · Tencent
LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models
Oct 19, 2026Desheng Xu, Hanglin Li, Tianchen Song +2LLM Quantization Post-Training Quantization
Li Auto Inc.
PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs
Oct 18, 2026Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang +1Memory-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning
University of Southern California
Do We Need the Largest LLMs for Every Clinical Decision?
Oct 17, 2026Ali H. Lazem, William TeahanLLM Quantization Clinical Decision Support
School of Computer Science and Engineering, Bangor University, Bangor, LL57 2DG, Gwynedd, United Kingdom. · University of Thi-Qar, Nasiriyah, 64001, Iraq.
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Oct 16, 2026Wei WangEfficient Transformer Inference GPU Kernel Optimization
Reusing Calibration Gains across Model Representations
Oct 16, 2026Xu Cao, Wenxuan XiaoPost-Hoc Calibration LLM Quantization
Astrmira Tech.
Noise-shaped Sampling and Annealing for Scalable On-the-edge Training of Binary and Ternary AI Models
Oct 15, 2026Zihao Chen, Zhili Xiao, Rui-Jie Zhu +3Binary Neural Networks Ternary Quantization
Department of Electrical and Systems Engineering, Washington University in St. Louis, One Brookings Drive, St. Louis, 63130, MO, USA. · Department of Electrical and Computer Engineering, University of California, Santa Cruz, 1156 High Street, Santa Cruz, 95064, CA, USA. · Department of Computer Science and Engineering, Washington University in St. Louis, One Brookings Drive, St. Louis, 63130, MO, USA.
N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models
Oct 15, 2026Donghoon Han, Minseop Jung, Dong Woo Kwak +2Memory Compression LLM Compression
Dnotitia Inc. · KAIST
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Oct 15, 2026Johannes Knittel, Alec Helbling, Hanspeter PfisterLLM Quantization Neural Network Quantization
Harvard University · Georgia Tech

Oct 27, 2026J/K move · Enter open · S save

CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models

Zukang Xu, Zhixiong Zhao, Xing Hu, Zhe Jiang+2

Houmo AI · Peking University · Southeast University

LLM Quantization Vector Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Large language models (LLMs) require substantial memory, motivating low-bit weight quantization for efficient deployment. Existing vector quantization (VQ) methods often improve accuracy by increasing the effective coding dimension, which can complicate independent lookup and local codeword updates. We propose CoS-VQ (Coordinated Shared-Scale Vector Quantization), which preserves low-dimensional codewords while jointly optimizing hard indices and their shared scale across positions. The key challenge is scale-induced coupling: the optimal scale depends on the entire codeword combination and, in turn, changes candidate rankings across positions, so pointwise updates can stall even after scale re-optimization. CoS-VQ collects candidates at multiple probe scales and constructs a tangent separable upper bound for the residual-aware quadratic objective. By reducing candidate comparisons to lower envelopes of lines over positive scales, it identifies finitely many representative intervals for joint index--scale updates without exhaustively enumerating codeword combinations. On Qwen3.8-27B, CoS-VQ outperforms QTIP and LLVQ on the evaluated language modeling, coding, and reasoning benchmarks. Ablations from the same initialization further show larger gains from joint updates than from scale-only refinement, highlighting joint selection as an alternative to increasing codeword dimensionality. The code is available at https://github.com/xzktx003/Cos_VQ.