Activation Quantization
Overview
Activation quantization studies how to represent the intermediate values passed between neural-network layers with fewer bits, reducing memory traffic and enabling faster, smaller deployments. Common approaches include calibration-based clipping and scale selection, smoothing or rotating activations to reduce outliers, mixed-precision allocation, and quantization-aware training; methods target formats such as INT8, INT4, and FP4. Work spans language and vision models, diffusion systems, speech models, and interactive policies, with evaluations using measures such as perplexity, ImageNet accuracy, generation quality, and task success. A central tension is preserving quality across changing activation distributions while delivering real speed and memory gains on the hardware and kernels used in deployment.
Latest papers
131 papers10 papers in the last four weeks, down 64% on the four weeks before. 0.1% of all new papers.
DyQ-VLA: Temporal-Dynamic-Aware Quantization for Embodied Vision-Language-Action Models
Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs
SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers
Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs
FreeAct: Freeing Activations for LLM Quantization
S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
Activation Quantization of Vision Encoders Needs Prefixing Registers
GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
basin'' to maintain generalization. Consequently, GPLQ employs a sequential activation-first, weights-later'' strategy. Stage 1 keeps weights in FP32 while quantizing activations with a feature mimicking loss in only 1 epoch to keep it stay in the same ``basin'', thereby preserving generalization. Stage 2 quantizes weights using a PTQ method. As a result, GPLQ is 100x faster than existing QAT methods, lowers memory footprint to levels even below FP32 training, and achieves 4-bit model performance that is highly competitive with FP32 models in terms of both accuracy on ImageNet and generalization to diverse downstream tasks, including fine-grained visual classification and object detection. We will release an easy-to-use open-source toolkit supporting multiple vision tasks.