LLM Quantization

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

33 papers in the last four weeks, up 43% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 251

All topics

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Sep 28, 2026Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei +2Mixed-Precision Quantization LLM Quantization
Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany +3
QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantization
Sep 28, 2026Qiulin Shang, Zhoutong Wu, Jie Hu +1LLM Quantization Large Language Model-Guided Optimization
Peking University
QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models
Sep 28, 2026Bang Hu, Guowei Zhu, Changze Lv +4LLM Quantization Spiking Neural Networks
School of Computer Science, Fudan University, Shanghai, China
Quantization Error Is Spectrally Flat: A Single Random Probe Is a Calibrated, Data-Free Sensitivity Estimator, with Application to Budget-Targeted Mixed-Precision Quantization
Sep 27, 2026I Kennedy, T KennedyMixed-Precision Quantization LLM Quantization
baa.ai Auckland, New Zealand
Softmax Reparameterization for Output-Head Quantization
Sep 25, 2026Asim Kadav, Christian Flores, Chirag Arora +5LLM Quantization Post-Training Quantization
Adobe Search, Discovery & ContentAI
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Sep 22, 2026Yuanteng Chen, Zhilei Liu, Peisong Wang +7LLM Quantization Quantization-Aware Training
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +2
Disaggregated Quantization: Specializing LLM Prefill and Decode
Sep 22, 2026Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi +2LLM Quantization LLM Inference
NVIDIA · ISTA
Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement
Sep 22, 2026Akihiro Yoshida, Yuma IchikawaMixed-Precision Quantization LLM Quantization
Fujitsu Limited, Institute of Science Tokyo · Fujitsu Limited, RIKEN Center for AIP
When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs
Sep 21, 2026Yeji Kim, Mi-Young Kim, Randy GoebelLLM Quantization Faithfulness of Language Model Explanations
University of Alberta
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
Sep 16, 2026Jerry KaplanLLM Quantization LLM Evaluation
Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University
The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?
Sep 15, 2026Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3LLM Inference Efficiency LLM Quantization
Vizuara
LLM Inference in a Flash!
Sep 14, 2026Sebastian Zhao, Minseo Kim, Coleman Hooper +5LLM Quantization Compute-in-Memory
UC Berkeley · ICSI · LBNL
Structured Transforms for Low-Overhead Quantization of Language Models
Sep 11, 2026Daria Cherniuk, Alexander Rudikov, Boris Kashin +1LLM Quantization Weight-Only Quantization
Institute of Numerical Mathematics · Steklov Mathematical Institute
Why Does Post-Training Quantization Work?
Sep 10, 2026Yuxiang Chen, Michael Beyer, Jun Zhu +1LLM Quantization Post-Training Quantization
Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany
Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution
Sep 10, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compression
Sep 7, 2026Dachi KurtskhaliaLLM Quantization Weight-Only Quantization
Impel
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Sep 3, 2026Sergii Kozyrev, Davyd MaiborodaLLM Quantization 4-Bit Quantization
Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
Sep 2, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
Sep 1, 2026Jundong Hu, Shekar RamachandranMixed-Precision Quantization LLM Quantization
PayPal AI
Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMs
Sep 1, 2026Jainil Dharmil ShahLLM Quantization LLM Evaluation
Elmore Family School of Electrical and Computer Engineering · Purdue University, West Lafayette, Indiana, USA
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
Aug 31, 2026Chun-Ting Chen, Dongmin Han, Hangyeol Mun +6LLM Quantization LLM Inference Acceleration
Cornell University New York, NY, USA · Intel Corporation Santa Clara, CA, USA · Intel Corporation Hillsboro, OR, USA
QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
Aug 31, 2026Yipin Guo, Arun M George, Jie Fu +3LLM Quantization Ternary Quantization
University of Notre Dame
Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs
Aug 31, 2026Deokjae Lee, Sihun Chu, Hyun Oh SongMixed-Precision Quantization LLM Quantization
Department of Computer Science and Engineering, Seoul National University · Neural Processing Research Center · 3Neural Processing Research Center +2
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
Aug 30, 2026Hongyu Yu, Yifei ShenLLM Quantization LLM Pruning
Lenovo Research · University of Washington
A Target-Centric Survey of Quantization-Aware Training
Aug 30, 2026Jiamin Song, Mengjie Zhao, Zijing Wang +6LLM Quantization Quantization-Aware Training
1Northeastern University, China · 2Shandong University, China · 3CIS, LMU Munich; MCML, Germany
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Mixed-Precision Quantization LLM Quantization
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Mixed-Precision Quantization LLM Quantization
South China University of Technology · Meituan · East China Normal University
When Personality Meets Quantization: A Layer-wise MBTI Analysis of Quantized LLMs
Aug 26, 2026Yao Fu, Lijia Huang, Xiaomin Li +3LLM Quantization LLM Interpretability
Case Western Reserve University Cleveland, OH 44106, USA · Northeastern University Boston, MA 02115, USA · Microsoft Research Redmond, WA 98052, USA
FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference
Aug 24, 2026Gongwei Lee, Ji Liu, Juncheng Jia +1Mixed-Precision Quantization LLM Quantization
School of Computer Science and Technology, Soochow University, Suzhou, China · Hithink Research, Hangzhou, China · Electronic Engineering, Tsinghua University, Beijing, China
Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair
Aug 24, 2026Zehao Liu, Chuangchuang Fang, Yang RenLLM Quantization Weight-Only Quantization
Huawei Technologies Research & Development (UK) Limited, Edinburgh, United Kingdom

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers

Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei, Milad Hosseini+1

Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany+3

Mixed-Precision Quantization LLM Quantization Attention Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Most post-training quantization pipelines fit each weight matrix to its pretrained counterpart, one matrix at a time. Whether that proxy tracks what an attention block actually computes, or how errors in the Q, K and V projections compound inside the softmax, is rarely checked. We write the objective on the attention output instead, over all three projections at once, and reuse it throughout the pipeline. JAB defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it twice: to fit the quantized weights (GPTQ warm start, then STE with learnable scales), and to score the block for a multiple-choice knapsack allocation. On attention-only quantization of Mistral-7B this works. At 3 bits JAB recovers 77-90% of the gap between uniform GPTQ and full precision, and its sensitivity estimate tracks an oracle costing 73 forward passes to within a fraction of a point. It stops working once MLP layers enter the allocation. A role-aware offset rule needing no sensitivity estimate at all beats JAB on GPT-2's MLP and on the full Mistral-7B model: with a 3-bit floor it quantizes 96.4% of the weights to 4.5 bits per parameter at 6.933 perplexity, within 4.4% of full precision (6.643) at 3.56x compression, against 7.158 for JAB at the same budget. Which matrix a weight sits in matters more than any sensitivity estimate we computed. Two things came out sideways. Block-local reconstruction is an unreliable proxy for end-to-end perplexity: one run improved a block's own objective 4.6x while perplexity rose 32x, which is why every allocation here is validated end-to-end. And on attention-only quantization, fine-tuning moved weights farther from their pretrained values while pulling attention outputs closer, with net gains. Post-training seems to recover attention behavior, not weights.

Momentum

Related topics

Latest papers 251