Overview

Weight-only quantization studies how to represent a model’s weights with fewer bits while preserving its behavior, especially when deploying large language models on memory-limited hardware. Methods range from rounding and group-wise post-training quantization to activation- or curvature-aware error correction, learned codebooks, rotations, and quantization-aware training; mixed precision can protect sensitive layers or weights. Work is evaluated using measures such as language-model perplexity, zero-shot and reasoning accuracy, generation quality, memory use, and throughput on target hardware. The main tension is that very low bit rates save substantial storage but can sharply damage capabilities, while better accuracy may require calibration, extra metadata, or kernels whose real speed depends on the hardware and workload.

Latest papers

CardsList

85 papers11 papers in the last four weeks, down 39% on the four weeks before. 0.1% of all new papers.

  1. LazyWeights: Decode-on-Demand Edge LLMs

    Nov 12, 2026Arnab Sanyal, Gourav Datta, Sandeep P. ChinchaliOn-Device Language Model InferenceLLM Compression

  2. What Does 4-Bit Microscaling Cost a Wearable Activity Recognizer?

    Nov 3, 2026Megan Healy, Hossein Valavi4-Bit QuantizationActivation Quantization

  3. SHq: Learnable Grids for LLM Weight Quantization

    Nov 2, 2026Michael Helcig, Earl Killian, Dan AlistarhLLM QuantizationNeural Network Quantization

  4. MXCode: Vector Quantization for MX-Native Inference at Scale

    Oct 30, 2026Gaurav Kolhe, Cansu Demirkiran, Shubhi Tyagi +8QuantizationLLM Quantization

  5. Quantization of Large Models with Weight Outliers and Sparsity

    Oct 30, 2026Leonidas Tsigkounakis, Abraham Jaeger Mountain, Yilang Zhang +1LLM QuantizationNeural Network Quantization

  6. FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization

    Oct 21, 2026Faraz Tahmasebi, Haocheng Xu, Andrew Trauger +4LLM Inference AccelerationLLM Quantization

  7. AGQ: Action-Guided Ultra-Low-Bit Quantization for Vision-Language-Action Models

    Oct 21, 2026Xuhong Qiu, Zhuoyu Wu, Qiawei Zheng +3VLM QuantizationEfficient VLA Model Inference

  8. RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

    Oct 15, 2026Johannes Knittel, Alec Helbling, Hanspeter PfisterLLM QuantizationNeural Network Quantization

  9. Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference

    Oct 13, 2026Petar BastaLLM Inference EfficiencyLLM Quantization

  10. FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization

    Oct 10, 2026Yujie Chen, Weiyan Zheng, Yiwu Yao +7Low-Bit QuantizationNeural Network Quantization

  11. SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

    Oct 10, 2026Xianglong Yan, Zenan Wu, Kewei Zhang +2LLM QuantizationWeight-Only Quantization

  12. When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

    Oct 8, 2026Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu +6LLM QuantizationPost-Training Quantization

  13. BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization

    Oct 3, 2026Chenhang Cui, Xu Xie, Linrui Xu +4LLM QuantizationLow-Bit Quantization

  14. JARQ: Joint Alternating Refinement for Quantization

    Sep 29, 2026Xinyu Wang, Sicheng Lyu, Xiao-Wen ChangLLM QuantizationWeight-Only Quantization

  15. ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights

    Sep 29, 2026Jonathan Mei, Sang Hyub Kim, Oliver Knitter +2LLM QuantizationWeight-Only Quantization

  16. Security-Enhanced Seed-Based Weight Quantization for Large Language Models

    Sep 29, 2026Qiuyu Ren, Sudipta Paria, Aritra Dasgupta +1LLM QuantizationLLM Compression

  17. Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights

    Sep 28, 2026Yiteng Peng, Zhibo Liu, Dongwei Xiao +1Mixed-Precision QuantizationEfficient Neural Network Inference

  18. PulseQuant: Propagation-Guided Subspace Correction for 4-Bit Video Diffusion Transformers

    Sep 27, 2026Yutong Wang, Xingtong Ge, Enhuai Liu +4Video Diffusion ModelsDiffusion Transformer

  19. Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement

    Sep 22, 2026Akihiro Yoshida, Yuma IchikawaMixed-Precision QuantizationLLM Quantization