Massive Activations

Follow this topic Ask about this topic

Latest papers

14 papers3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Massive Activation Gating Channel in Large Language Models
Oct 6, 2026Minjia Mao, Shi Chen, Bowen Yin +1Massive Activations LLM Interpretability
University of Delaware · Peking University
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight Decay Attention Sinks
Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA
Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers
Sep 28, 2026Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1Transformer Interpretability Transformer FFNs
PsiDagger · IIT Delhi
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir UddinLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Aug 12, 2026Zunhai Su, Bohan Sun, Xialie Zhuang +9Linear Attention Hybrid Attention
Startlux · Tsinghua University · University of Chinese Academy of Sciences +1
Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation
Jul 3, 2026Chaofan Gan, Zicheng Zhao, Yuanpeng Tu +6Visual Representation Learning Diffusion Transformer
Shanghai Jiao Tong University, Shanghai, China · The University of Hong Kong, Hong Kong, China · Monash University, Melbourne, Australia +2
Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test
Jun 17, 2026Maruthi VemulaTransformer Massive Activations
University of North Carolina at Chapel Hill
Measuring Maximum Activations in Open Large Language Models
May 15, 2026Luxuan Chen, Han Tian, Xinran Chen +9LLM Quantization Activation Outliers
1Shanghai Jiao Tong University · 2Baidu Inc. · 4Nankai University +1
Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers
May 13, 2026Evelyn Turri, Davide Bucciarelli, Sara Sarto +2Diffusion Transformer Massive Activations
University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
May 8, 2026Zeru Shi, Zhenting Wang, Fan Yang +2LLM Interpretability Attention Sinks
Rutgers University · Wake Forest University · Meta AI
Steering Video Diffusion Transformers with Massive Activations
Mar 18, 2026Xianhang Cheng, Yujian Zheng, Zhenyu Xie +2Massive Activations Video Diffusion Models
MBZUAI · Pinscreen
Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
Mar 18, 2026Yihong Chen, Zhouchen Lin, Quanming YaoMassive Activations Attention Sinks
Tsinghua University · Peking University
The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
Mar 5, 2026Shangwen Sun, Alfredo Canziani, Yann LeCun +1Massive Activations Attention Sinks
New York University.
Probing Graph Neural Network Activation Patterns Through Graph Topology
Feb 24, 2026Floriano Tori, Lorenzo Bini, Marco Sorbi +2Massive Activations Graph Transformer
Business Technology and Operations, Vrije Universiteit Brussels, Brussels, Belgium · Department of Computer Science, University of Geneva, Geneva, Switzerland · Research Institute for Statistics and Information Science, Geneva, Switzerland +1

Oct 6, 2026J/K move · Enter open · S save

Massive Activation Gating Channel in Large Language Models

Minjia Mao, Shi Chen, Bowen Yin, Xiao Fang

University of Delaware · Peking University

Massive Activations LLM Interpretability

Open paper AskPDF arXiv

Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mechanism by which a token develops massive activations as it propagates through a pretrained LLM remains poorly understood. In this paper, we find that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN). The position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. We further provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations.