Low-Bit Quantization

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 62% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 155

All topics

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
Oct 7, 2026Sunjoo Whang, Jungjun Oh, Minsung Kim +5KV-Cache Compression KV-Cache Quantization
KAIST · GIST · Google Research
CurveTQ: Rotation-Free Trellis Quantization of LLM Weights via Curvature-Weighted Search
Oct 6, 2026Guanhua Ding, Zi Wang, Ruichao Li +1LLM Quantization Neural Network Quantization
Black Sesame Technologies
Few Bits, One Law: Toward W2A4KV2
Oct 6, 2026Kai Yi, Tarek Elgamal, Sruthikesh Surineni +3LLM Quantization Low-Bit Quantization
Meta AI
Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models
Oct 6, 2026Seobin Song, Geonho Lee, Janghwan Lee +1LLM Quantization Post-Training Quantization
Hanyang University, Seoul, Republic of Korea · Qualcomm AI Research, Qualcomm Korea YH, Seoul, Republic of Korea
ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization
Oct 6, 2026Aksel Fristrup, Sumit Pandey, Ankit KariryaaLLM Quantization Rotation-Based Quantization
Henselian (henselian.com) · Department of Computer Science University of Copenhagen Denmark · Towards Deep Learning
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
Oct 5, 2026Zhe Wei, Mengqi Guo, Yuan Yuan +3Mixed-Precision Quantization LLM Quantization
Huawei Technologies Ltd.
Loopy: Low-Bit Quantization Framework for Looped Language Models
Oct 4, 2026Zeyu LI, Yipu ZHANG, Jintao Chen +2LLM Quantization Post-Training Quantization
The Hong Kong University of Science and Technology · Duke Kunshan University
Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2
Oct 1, 2026Yohan Chatelain, Pablo de Oliveira CastroTransformer Inference Low-Bit Quantization
Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada · Université Paris-Saclay, UVSQ, LI-PaRAD Versailles, France · Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France
Q-MINO: A Minimal-Norm Method for Quantization-Aware Training
Sep 30, 2026Don LiQuantization-Aware Training Low-Bit Quantization
Department of Mathematics & Statistics Portland State University
XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantization
Sep 30, 2026Xiaofan Que, Nir Elkayam, Spandan Pyakurel +2LLM Quantization LLM Inference Acceleration
Arm, Inc. · Rochester Institute of Technology
Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication
Sep 30, 2026Shuxiao Xie, Shuyang Xie, Yuan Cao +3LLM Inference Acceleration Low-Bit Quantization
ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights
Sep 29, 2026Jonathan Mei, Sang Hyub Kim, Oliver Knitter +2LLM Quantization Weight-Only Quantization
Applications R&D IonQ College Park, MD 20740, USA
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs
Sep 28, 2026Mehdi Makni, Ryan Lucas, Rahul MazumderLLM Quantization Rotation-Based Quantization
Operations Research Center Massachusetts Institute of Technology
TORQUE: Optimizing What (not) to Quantize Before and After Rotation
Sep 28, 2026Ran Ben Basat, Michael Mitzenmacher, Shay VargaftikRotation-Based Quantization Vector Quantization
University College London · Harvard University · VMware Research by Broadcom
Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter
Sep 28, 2026Pier-Jean MalandrinoLLM Quantization LLM Inference Acceleration
Scub, Bordeaux, France
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
Sep 28, 2026Peilin Sun, Guang Liang, Jin Tong +1Post-Training Quantization Low-Bit Quantization
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Sep 22, 2026Yuanteng Chen, Zhilei Liu, Peisong Wang +7LLM Quantization Quantization-Aware Training
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +2
QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models
Sep 22, 2026Jiaqi Zhao, Xiaobin Hu, Bo Yin +3KV-Cache Compression KV-Cache Quantization
Harbin Institute of Technology (Shenzhen), Shenzhen, China · National University of Singapore, Singapore
RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models
Sep 21, 2026Zhiping Wu, Dongdong Ren, Yangchengyu Zhou +4VLM Quantization Large Vision-Language Models
State Key Laboratory for Novel Software Technology and the School of Intelligence Science and Technology, Nanjing University, Suzhou Campus, Suzhou 215163, China
SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models
Sep 21, 2026Kewei Zhang, Zheng Chen, Haotong Qin +1VLM Quantization Efficient VLM Inference
Shanghai Jiao Tong University · The Hong Kong Polytechnic University
MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration
Sep 17, 2026Yuan Liao, Jae-sun SeoAI Accelerator Inference VLM Quantization
Cornell Tech, Cornell University New York, NY, USA
Predict Before You Deploy: Offline Prediction of Quantization-Induced Task Degradation for World Action Models
Sep 16, 2026Jiuyi Xu, Jinjia Guo, Meida Chen +2World Action Models Post-Training Quantization
Robotics Program, Colorado School of Mines, Golden, CO USA. · Department of Civil and Coastal Engineering, University of Florida, Gainesville, FL USA. · University of Southern California - Institute for Creative Technology, Los Angeles, CA USA. +1
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
Sep 14, 2026Xingyang Li, Dongyun Zou, Shining Zhang +8Video Diffusion Models Diffusion Transformer
2MIT · 1Nunchux AI · 3CMU +3
Structured Transforms for Low-Overhead Quantization of Language Models
Sep 11, 2026Daria Cherniuk, Alexander Rudikov, Boris Kashin +1LLM Quantization Weight-Only Quantization
Institute of Numerical Mathematics · Steklov Mathematical Institute
When Does Low-Bit Quantization Preserve the Decisions of Vector Search?
Sep 9, 2026Wenxuan Xiao, Xu CaoVector Quantization Low-Bit Quantization
Astrmira Tech.
EFQ-Softmax: Exp-Free Quantization for Softmax
Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Efficient Transformer Inference Transformer Attention
Xi'an Jiaotong University · Xi’an Jiaotong University · Huawei Technologies Co., Ltd +1
SCULPT: Training Edge Vision Models for Post-Training Quantization Readiness
Sep 1, 2026Bharadwaj Kavuri, Sourav Babu-PK, Varadhraj Ellapan +2Edge Computing Activation Quantization
Valeo Vision Systems, Tuam, Ireland · Valeo India Pvt. Ltd, Chennai, India
QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
Aug 31, 2026Yipin Guo, Arun M George, Jie Fu +3LLM Quantization Ternary Quantization
University of Notre Dame

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han+2

KAIST · Yonsei University · Seoul National University

KV-Cache Quantization Efficient Transformer Inference Low-Bit Quantization Recurrent Transformers

Open paper AskPDF arXiv

Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count. However, KV cache memory still scales with the number of loops, becoming a key memory bottleneck that limits batch size and inference throughput. KV cache quantization can alleviate this bottleneck, but existing methods often suffer substantial accuracy degradation at aggressive low-precision regimes. We observe that looped Transformers offer a unique opportunity: KV states across loops are highly similar. Based on this observation, we propose ResidualQuant, which uses the final-loop KV states as a reference and represents the remaining loops with low-precision residuals. Our method further combines least-square scaling and rotations applied to the residuals, as well as loop-wise mixed precision, to enable accurate quantization down to INT2 while retaining efficient reconstruction. Across multiple looped Transformer models and mathematical reasoning and code generation benchmarks, ResidualQuant consistently improves the accuracy-memory tradeoff over state-of-the-art rotation-based KV quantization. In particular, our method retains accuracy close to BF16 under mixed-precision settings while reducing theoretical KV storage by 80.7%, achieving up to 13.0% higher accuracy than the rotation-based baseline at the same memory budget. On an RTX 5090, the reduced KV memory traffic improves fixed-batch decode throughput by up to 2.73x, while the smaller memory footprint enables up to 2x larger batches, improving peak throughput by up to 4.15x.

Momentum

Related topics

Latest papers 155