Weight-Only Quantization

Ask about this topic

Overview

Weight-only quantization studies how to represent a model’s weights with fewer bits while preserving its behavior, especially when deploying large language models on memory-limited hardware. Methods range from rounding and group-wise post-training quantization to activation- or curvature-aware error correction, learned codebooks, rotations, and quantization-aware training; mixed precision can protect sensitive layers or weights. Work is evaluated using measures such as language-model perplexity, zero-shot and reasoning accuracy, generation quality, memory use, and throughput on target hardware. The main tension is that very low bit rates save substantial storage but can sharply damage capabilities, while better accuracy may require calibration, extra metadata, or kernels whose real speed depends on the hardware and workload.

Latest papers

CardsList

76 papers18 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

  1. A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models

    May 14, 2026Earl KillianMixed-Precision QuantizationLLM Quantization

  2. High-Rate Quantized Matrix Multiplication II

    May 13, 2026Or Ordentlich, Yury PolyanskiyLLM QuantizationRate-Distortion Theory

    the Hebrew University of Jerusalem, Israel · MIT, USA

  3. LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss

    May 9, 2026Euntae Choi, Sumin Song, Sungjoo YooLLM QuantizationQuantization-Aware Training

    Seoul National University

  4. AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

    May 9, 2026Beshr IslamBouli, David JinLLM Quantization4-Bit Quantization

    Massachusetts Institute of Technology, Cambridge, MA, USA.

  5. OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

    May 6, 2026Zhikai Li, Zhen Dong, Xuewen Liu +2LLM QuantizationWeight-Only Quantization

    Institute of Automation, Chinese Academy of Sciences · University of California, Santa Barbara.

  6. Rethinking Residual Errors in Compensation-based LLM Quantization

    Apr 9, 2026Shuaiting Li, Juncan Deng, Kedong Xu +5Quantization Error CompensationLLM Quantization

    Zhejiang University · vivo Mobile Communication Co., Ltd

  7. Weight Group-wise Post-Training Quantization for Medical Foundation Model

    Apr 9, 2026Yineng Chen, Peng Huang, Aozhong Zhang +9Post-Training QuantizationLow-Bit Quantization

    University at Albany, SUNY · Southwest Jiaotong University · Purdue University +1

  8. PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

    Mar 30, 2026Caio VicentinoLLM QuantizationNeural Network Quantization

  9. ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing

    Mar 30, 2026Edward J. YoonLLM QuantizationTernary Quantization

  10. SliderQuant: Accurate Post-Training Quantization for LLMs

    Mar 26, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +3LLM QuantizationPost-Training Quantization

    Intel Labs China · BUPT

  11. DAQ: Delta-Aware Quantization for Post-Training LLM Weight Compression

    Mar 20, 2026Xiaoming Yu, Shize Tang, Guanghua Yu +4QuantizationLLM Quantization

    Yuanbao & Hunyuan AI Infra Team

  12. WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

    Mar 5, 2026Egor Lifar, Semyon Savkin, Or Ordentlich +1Neural Network QuantizationLLM Quantization

    MIT · Independent Researcher · Hebrew University of Jerusalem

  13. 1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization

    Feb 17, 2026Sohir Maskey, Constantin Eichenberg, Johannes Messner +1QuantizationLLM Quantization

    Aleph Alpha Research · Graphcore Research

  14. Activation Sensitivity as a Unifying Principle for Post-Training Quantization

    Jan 15, 2026Bruce Changlong XuLLM QuantizationPost-Training Quantization

  15. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    Nov 9, 2025Amit LeVi, Raz Lapid, Rom Himelstein +3Weight-Only QuantizationMixed-Precision Quantization

    1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1

  16. Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

    Jul 24, 2025Qingcheng Zhu, Yangyang Ren, Linlin Yang +6Mixed-Precision QuantizationLLM Quantization

    Beihang University · Communication University of China · Beijing Jiaotong University