Activation Outliers

Follow this topic Ask about this topic

Activation outliers are unusually large values, channels, or tokens in a model’s internal representations; they can dominate quantization scales, increase low-bit errors, and sometimes disrupt feature learning. Researchers study their causes, including channel structure, coherent mean bias, residual streams, and attention sinks, and measure their effects across language models, vision transformers, and diffusion models. Mitigations include clipping or compensating values, channel-aware precision, rotations and offsets, register tokens, and training-time regularization that reshapes activations. Results show that low-bit deployment can improve, but a central tension is how to control harmful outliers without removing signals the model needs, across different architectures and hardware.

Latest papers

17 papers0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
Jun 5, 2026Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang +2LLM Quantization 4-Bit Quantization
School of Computer and Communication Sciences, EPFL, Switzerland · 2Huawei, Switzerland · 3Swiss Data Science Center, ETHZ & EPFL, Switzerland
Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
Jun 1, 2026Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou +1LLM Quantization Activation Outliers
Princeton University, NJ, USA · EnCharge AI, CA, USA
On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders
May 29, 2026Elana Simon, Etowah Adams, James ZouActivation Outliers Sparse Autoencoders
1Stanford University · 2Columbia University
The Quantization Benefits of Residual-Free Transformers
May 25, 2026Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2Transformer Activation Outliers
Australian Institute for Machine Learning, Adelaide University · DATA61, CSIRO
Attention Sinks and Outliers in Attention Residuals
May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM Quantization Attention Residual Connections
Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5
Bug or Feature $^2$ : Weight Drift and Activation Sparsity
May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural Networks Activation Sparsity
Measuring Maximum Activations in Open Large Language Models
May 15, 2026Luxuan Chen, Han Tian, Xinran Chen +9LLM Quantization Activation Outliers
1Shanghai Jiao Tong University · 2Baidu Inc. · 4Nankai University +1
Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantization
May 14, 2026Peilin Sun, Jianxin WuActivation Outliers Quantization Error Compensation
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Taming Outlier Tokens in Diffusion Transformers
May 6, 2026Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu +3Diffusion Transformer Activation Outliers
Rice University · Apple
Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
May 2, 2026Jin Tong, Guang Liang, Peilin Sun +1Vision Transformer Activation Outliers
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels
Apr 27, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaLLM Quantization Activation Outliers
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Apr 20, 2026Ziyang LiuLLM Quantization 4-Bit Quantization
Independent Researcher
Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs
Apr 9, 2026Binxing Xu, Hao Gu, Lujun Li +8LLM Quantization Quantization-Aware Training
Zhejiang University · Hong Kong University of Science and Technology · City University of Hong Kong
The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Mar 11, 2026Hengjie Cao, Zhendong Huang, Mengyi Chen +15LLM Quantization 4-Bit Quantization
Fudan University · Shanghai Innovation Institute · University of Bath +5
Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs
Mar 4, 2026Pranav Kumar KaliaperumalActivation Outliers Post-Training Quantization
M.S. Computer Science, University of Colorado Denver, Aurora, Colorado, USA
S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
Feb 16, 2026Arnav Chavan, Nahush Lele, Udbhav Bamba +3Activation Outliers Activation Quantization
Amazon · Carnegie Mellon University
Activation Quantization of Vision Encoders Needs Prefixing Registers
Oct 6, 2025Seunghyeon Kim, Taesun Yeom, Jinho Kim +3Activation Outliers Activation Quantization
POSTECH · NYU Langone Health2 · Google3

Jun 5, 2026J/K move · Enter open · S save

OffQ: Taming Structured Outliers in LLM Quantization by Offsetting

Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang, Mathieu Salzmann+1

School of Computer and Communication Sciences, EPFL, Switzerland · 2Huawei, Switzerland · 3Swiss Data Science Center, ETHZ & EPFL, Switzerland

LLM Quantization 4-Bit Quantization Activation Outliers Activation Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Low-bit quantization has been widely adopted to accelerate the inference of large language models (LLMs) by significantly reducing computational cost and memory usage. However, activation outliers pose a major challenge to effective quantization, often leading to notable performance degradation. In this paper, we introduce OffQ, a method designed to mitigate activation outliers in low-bit quantization through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a proposed top-1 PCA, and then concentrates high-magnitude activations into 1 channel via rotation. OffQ then absorbs this concentrated outlier channel by converting its magnitude into a shared offset, thereby reducing the standard deviation of the activations. This offsetting strategy enables effective W4A4KV4 quantization of LLMs using deployment-friendly uniform-grid and uniform-precision quantization. Extensive experiments across diverse LLM architectures and benchmarks demonstrate that OffQ outperforms state-of-the-art baselines, consistently improving model accuracy while preserving low-bit efficiency.