Activation Outliers

Follow this topic Ask about this topic

Momentum

0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 13

All topics

OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
Jun 5, 2026Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang +2LLM Quantization 4-Bit Quantization
School of Computer and Communication Sciences, EPFL, Switzerland · 2Huawei, Switzerland · 3Swiss Data Science Center, ETHZ & EPFL, Switzerland
Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
Jun 1, 2026Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou +1LLM Quantization Activation Outliers
Princeton University, NJ, USA · EnCharge AI, CA, USA
On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders
May 29, 2026Elana Simon, Etowah Adams, James ZouActivation Outliers Sparse Autoencoders
1Stanford University · 2Columbia University
The Quantization Benefits of Residual-Free Transformers
May 25, 2026Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2Transformer Activation Outliers
Australian Institute for Machine Learning, Adelaide University · DATA61, CSIRO
Attention Sinks and Outliers in Attention Residuals
May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM Quantization Attention Residual Connections
Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5
Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes
May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural Networks Activation Sparsity
Measuring Maximum Activations in Open Large Language Models
May 15, 2026Luxuan Chen, Han Tian, Xinran Chen +9LLM Quantization Activation Outliers
1Shanghai Jiao Tong University · 2Baidu Inc. · 4Nankai University +1
Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantization
May 14, 2026Peilin Sun, Jianxin WuActivation Outliers Post-Training Quantization
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Taming Outlier Tokens in Diffusion Transformers
May 6, 2026Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu +3Diffusion Transformer Activation Outliers
Rice University · Apple
Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
May 2, 2026Jin Tong, Guang Liang, Peilin Sun +1Vision Transformer Activation Outliers
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels
Apr 27, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaLLM Quantization Activation Outliers
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Apr 20, 2026Ziyang LiuLLM Quantization 4-Bit Quantization
Independent Researcher
Activation Quantization of Vision Encoders Needs Prefixing Registers
Oct 6, 2025Seunghyeon Kim, Taesun Yeom, Jinho Kim +3Activation Outliers Activation Quantization
POSTECH · NYU Langone Health2 · Google3

Jun 5, 2026, cs.LGJ/K move · Enter open · S save

OffQ: Taming Structured Outliers in LLM Quantization by Offsetting

Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang, Mathieu Salzmann+1

School of Computer and Communication Sciences, EPFL, Switzerland · 2Huawei, Switzerland · 3Swiss Data Science Center, ETHZ & EPFL, Switzerland

LLM Quantization 4-Bit Quantization Activation Outliers Activation Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Low-bit quantization has been widely adopted to accelerate the inference of large language models (LLMs) by significantly reducing computational cost and memory usage. However, activation outliers pose a major challenge to effective quantization, often leading to notable performance degradation. In this paper, we introduce OffQ, a method designed to mitigate activation outliers in low-bit quantization through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a proposed top-1 PCA, and then concentrates high-magnitude activations into 1 channel via rotation. OffQ then absorbs this concentrated outlier channel by converting its magnitude into a shared offset, thereby reducing the standard deviation of the activations. This offsetting strategy enables effective W4A4KV4 quantization of LLMs using deployment-friendly uniform-grid and uniform-precision quantization. Extensive experiments across diverse LLM architectures and benchmarks demonstrate that OffQ outperforms state-of-the-art baselines, consistently improving model accuracy while preserving low-bit efficiency.

Momentum

Related topics

Latest papers 13