Activation Quantization

Ask about this topic

Overview

Activation quantization studies how to represent the intermediate values passed between neural-network layers with fewer bits, reducing memory traffic and enabling faster, smaller deployments. Common approaches include calibration-based clipping and scale selection, smoothing or rotating activations to reduce outliers, mixed-precision allocation, and quantization-aware training; methods target formats such as INT8, INT4, and FP4. Work spans language and vision models, diffusion systems, speech models, and interactive policies, with evaluations using measures such as perplexity, ImageNet accuracy, generation quality, and task success. A central tension is preserving quality across changing activation distributions while delivering real speed and memory gains on the hardware and kernels used in deployment.

Latest papers

CardsList

120 papers24 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

  1. QB-LIF: Learnable-Scale Quantized Burst Neurons for Efficient SNNs

    Apr 28, 2026Dewei Bai, Hongxiang Peng, Jiajun Mei +4Efficient Neural Network InferenceSpiking Neural Networks

  2. LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation

    Apr 21, 2026Siqing Song, Chuang Wang, Yong Lang +2LLM QuantizationActivation Quantization

    MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei

  3. AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantization

    Apr 20, 2026Kosuke Matsushima, Yasuyuki Okoshi, Masato Motomura +1Compute-in-MemoryLong-Context Language Model Inference

    Institute of Science Tokyo†

  4. Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition

    Apr 20, 2026Ziyang LiuLLM Quantization4-Bit Quantization

    Independent Researcher

  5. DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization

    Apr 20, 2026Haokun Lin, Xinle Jia, Haobo Xu +7LLM Quantization4-Bit Quantization

    CASIA · CityU · NJU +3

  6. When W4A4 Breaks Camouflaged Object Detection: Token-Group Dual-Constraint Activation Quantization

    Apr 18, 2026Tianqi Li, Wenyu Fang, Xin He +34-Bit QuantizationActivation Quantization

    VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · Institute for Infocomm Research, A*STAR +2

  7. OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension

    Apr 14, 2026Zhiyuan Zhang, Yanzhao Li, Zhiqiang Zou +4LLM QuantizationNeural Network Quantization

    Huawei Technology

  8. ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation

    Apr 13, 2026Suyoung Kim, Sunghyun Wee, Hyeonjin Kim +3LLM QuantizationPost-Training Quantization

    Seoul National University, Seoul, Republic of Korea · LG Electronics, Seoul, Republic of Korea

  9. Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

    Apr 9, 2026Binxing Xu, Hao Gu, Lujun Li +8LLM QuantizationQuantization-Aware Training

    Zhejiang University · Hong Kong University of Science and Technology · City University of Hong Kong

  10. SL-FAC: A Communication-Efficient Split Learning Framework with Frequency-Aware Compression

    Apr 8, 2026Zehang Lin, Miao Yang, Haihan Zhu +9Activation CompressionCommunication-Efficient Distributed Training

    School of Computer and Information Engineering, Xiamen University of Technology, Xiamen, 361000, China · School of Artificial Intelligence, Hebei University of Technology · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong +5

  11. MUXQ: Mixed-to-Uniform Precision MatriX Quantization via Low-Rank Outlier Decomposition

    Apr 6, 2026Seoungsub Lee, In Seo Kim, Seon Wook KimLLM QuantizationNeural Network Quantization

    School of Electrical Engineering, Korea University, Seoul, Korea

  12. RUQuant: Towards Refining Uniform Quantization for Large Language Models

    Apr 5, 2026Han Liu, Haotian Gao, Changya Li +4LLM QuantizationActivation Quantization

    Dalian University of Technology Dalian, China · Peking University Beijing, China · Macao Polytechnic University Macao, China

  13. BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

    Apr 5, 2026Yifu Ding, Xianglong Liu, Shenghao Jin +2Binary QuantizationLow-Bit Quantization

    School of Computer Science and Engineering, Beihang University, Beijing, China. · Institute of Artificial Intelligence, Beihang University, Beijing, China. · Department of Automation, Tsinghua University, Beijing, China.

  14. SliderQuant: Accurate Post-Training Quantization for LLMs

    Mar 26, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +3LLM QuantizationPost-Training Quantization

    Intel Labs China · BUPT

  15. Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition

    Mar 20, 2026Calvin Ang, Sungyoon Kim, Mert PilanciNeural Network QuantizationActivation Quantization

    Stanford University

  16. DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

    Mar 19, 2026Maoyang Xiang, Bo WangNeural Network Activation FunctionsEfficient Transformer Inference

    Information Systems Technology and Design Singapore University of Technology and Design Singapore

  17. The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training

    Mar 11, 2026Hengjie Cao, Zhendong Huang, Mengyi Chen +15LLM Quantization4-Bit Quantization

    Fudan University · Shanghai Innovation Institute · University of Bath +5

  18. TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

    Mar 11, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangLLM QuantizationNeural Network Quantization

    Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139, USA

  19. SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

    Mar 9, 2026Yeonsik Park, Hyeonseong Kim, Seungkyu ChoiLLM QuantizationPost-Training Quantization

    Kyung Hee University · Yonsei University

  20. Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models

    Mar 9, 2026Lucas RakotoarivonyLow-Bit QuantizationActivation Quantization

    Thales, cortAIx Labs, 1 Av. Augustin Fresnel, 91120 Palaiseau, France

  21. DyQ-VLA: Temporal-Dynamic-Aware Quantization for Embodied Vision-Language-Action Models

    Mar 9, 2026Zihao Zheng, Hangyu Cao, Chibang Tao +9VLM QuantizationEfficient VLA Model Inference

    School of Computer Science, Peking University, Beijing, China. · School of Software Engineering, South China University of Technology, Guangzhou, China. · School of Artificial Intelligence, Beijing Normal University, Beijing, China. +1

  22. Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

    Mar 4, 2026Pranav Kumar KaliaperumalActivation OutliersPost-Training Quantization

    M.S. Computer Science, University of Colorado Denver, Aurora, Colorado, USA

  23. SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers

    Mar 3, 2026Wonsuk Jang, Thierry TambeDiffusion Model QuantizationNeural Network Quantization

    Stanford University

  24. Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs

    Mar 3, 2026Wuyue Zhang, Chongdong Huang, Chunbo You +3Low-Precision TrainingLLM Training

    Zhejiang Lab, Hangzhou, China

  25. FreeAct: Freeing Activations for LLM Quantization

    Mar 2, 2026Xiaohao Liu, Xiaobo Xia, Manyi Zhang +6LLM QuantizationActivation Quantization

    National University of Singapore · Huawei Technology · Central South University

  26. S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations

    Feb 16, 2026Arnav Chavan, Nahush Lele, Udbhav Bamba +3Activation OutliersActivation Quantization

    Amazon · Carnegie Mellon University

  27. Activation Quantization of Vision Encoders Needs Prefixing Registers

    Oct 6, 2025Seunghyeon Kim, Taesun Yeom, Jinho Kim +3Activation OutliersActivation Quantization

    POSTECH · NYU Langone Health2 · Google3

  28. GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

    Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training4-Bit Quantization

    State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China

  29. FPTQuant: Function-Preserving Transforms for LLM Quantization

    Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM QuantizationLLM Inference Acceleration

    Qualcomm AI Research, Amsterdam, Netherlands

  30. On Orthogonal Recurrent Neural Networks Quantization

    Feb 5, 2024Armand Foucault, Franck Mamalet, François MalgouyresRecurrent Neural NetworksNeural Network Quantization

    IMT · Comue de Toulouse