LLM Quantization

LLM: Large Language Model

Overview

LLM quantization studies how to represent model weights, activations, and attention key-value caches with fewer bits while preserving useful behavior and making deployment more efficient. Post-training methods such as GPTQ and AWQ use calibration data to round weights, while quantization-aware training, mixed-precision allocation, rotations, and outlier handling aim to reduce errors that accumulate across layers. Work also co-designs number formats and GPU, CPU, and edge-device kernels, evaluating trade-offs with language-model perplexity, zero-shot and reasoning benchmarks, memory, latency, and energy. Four-bit inference is often practical, but performance at very low bit widths remains model- and task-dependent, and an open challenge is ensuring that gains in aggregate accuracy do not hide losses in reasoning, safety, fairness, or uncertainty.

Latest papers

CardsList

443 papers68 papers in the last four weeks, down 13% on the four weeks before. 0.5% of all new papers.

  1. HADVQ: Hessian Aware Directional Vector Quantization

    Nov 17, 2026Daniil Li, Mars Davletshin, Vadim Klyzhenko +4QuantizationLLM Quantization

  2. SignRot: LLM Quantization with Massive Outlier-Aware Sign-Adjusted Rotation

    Nov 12, 2026Jaehun Gim, Jaewoo Kim, Gyuwan Kim +1LLM QuantizationNeural Network Quantization

  3. SoScale: Second-Order Scale Search for FP4 Quantization

    Nov 12, 2026Pengxiang Zhao, Xing Li, Hui-Ling Zhen +3LLM QuantizationNeural Network Quantization

  4. Gradient-Guided Hardware-Native MIXFP4

    Nov 12, 2026Jian-Jia Chen, Brian J. Chan, Wen-Hung Lee +8QuantizationLLM Quantization

  5. SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering

    Nov 10, 2026Genming Cui, Xiangyu Li, Zixu Hao +3LLM QuantizationQuantization-Aware Distillation

  6. AutoQS: Automated Quantization Strategy Design with Large Language Models

    Nov 9, 2026Zhen Liu, Zhiqin Yang, Yonggang Zhang +3LLM QuantizationQuantization

  7. Language model capabilities fail in a shared, predictable order under compression and perturbation

    Nov 8, 2026Zhixiang Wang, Ulas BagciLanguage Model RobustnessLanguage Model Performance Prediction

  8. When Calibration Data Matter in Sub-4-Bit Language Model Quantization

    Nov 8, 2026Yehong Jiang, Yen-Kuang Chen, Xinmin TianLLM QuantizationPost-Training Quantization

  9. MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations

    Nov 7, 2026Lorenz K. Muller, Philippe Bich, Chiara Boretti +3Post-Training QuantizationNeural Network Quantization

  10. AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement

    Nov 6, 2026Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan +4LLM QuantizationMixed-Precision Quantization

  11. CodeQuant2: Enhanced Outlier Mitigation and Adaptive Clustering for Low-Precision LRMs

    Nov 6, 2026Xingyu Liu, Xiangyang Yin, Tianhua Xia +2LLM QuantizationNeural Network Quantization

  12. ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training

    Nov 6, 2026Liqun Ma, Shengkun Tang, Liangyu Wang +7Optimizer State CompressionLLM Quantization

  13. StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

    Nov 5, 2026Jake Hyun, Ziyu Zhao, Jae-sun Seo +1LLM InferenceLLM Inference Efficiency

  14. ProQuant: Progressive Quantization-aware Training for Edge MLLMs

    Nov 5, 2026Yufei Xue, Yushi Huang, Jiawei Shao +5LLM QuantizationQuantization-Aware Training

  15. Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs

    Nov 5, 2026Md. Rafiul Biswas, George Mikros, Wajdi Zaghouanilegal language model evaluationLLM Compression

  16. ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference

    Nov 5, 2026Bokun Wang, Hung-Yueh Chiang, Diana MarculescuLLM InferenceEfficient Language Model Inference