Activation Sparsity

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 40

All topics

TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference
Oct 1, 2026Mukund Agarwalla, Chih-Jen LinActivation Sparsity LLM Inference Acceleration
MBZUAI · National Taiwan University
Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability
Sep 29, 2026Zhenting Huang, Bo Jiang, Junnan Liu +2Activation Sparsity LLM Interpretability
Beihang University · Monash University · Zhongguancun Laboratory
Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
Sep 27, 2026Jungseob Lee, Seungyoon Lee, Seongtae Hong +2Language Model Decoding Activation Sparsity
Korea University · Yonsei University Mirae Campus
SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking
Sep 21, 2026Pengzhi Zhong, Jiwei Mo, Haolun Li +4Activation Sparsity Visual Object Tracking
College of Computer Science and Engineering, Guilin University of Technology, Guilin 541004, China · School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Sep 8, 2026Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2Neural Network Generalization ReLU Neural Networks
University of New South Wales · University of Texas at Austin · University of Sydney
SPARK: Input-Conditioned Sparse Activation Modulation for Frozen DiT-based Super-Resolution
Sep 3, 2026Federico Putamorsi, Leonardo Zini, Marcella Cornia +1Diffusion Models for Image Restoration Diffusion Transformer
University of Modena and Reggio Emilia, Italy
Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
Aug 31, 2026Simon Richter, Ruhai Lin, Jason Yik +4Activation Sparsity LLM Inference Acceleration
Department of Electrical and Computer Engineering, Aarhus University, Aarhus, Denmark · Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, USA · School of Engineering and Applied Sciences, Harvard University, Cambridge, USA +1
Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Neuromorphic Virtual Sensing
Aug 25, 2026William Howes, Farid Ahmed, Syed Bahauddin AlamActivation Sparsity Neuromorphic Computing
Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign, Urbana, IL, USA · National Center for Supercomputing Applications, Urbana, IL, USA
Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Activation Sparsity Structured Sparsity
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications
The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy
Jul 29, 2026Zeyu WangEfficient Neural Network Inference Spiking Transformers
Georgia Institute of Technology
At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference
Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Efficient Neural Network Inference Transformer Inference
*IIS, ETH Zürich, Switzerland · ‡Computing Systems Lab, Huawei, Switzerland · †DEI, Universit`a di Bologna, Italy
Neural Feature Governance: Extending Atom Prevalence
Jul 23, 2026Idris Karel Seunda Ekwe, Patrick Tenga Shako, Ernest Parfait FokouéBayesian Neural Networks Uncertainty Quantification
African Institute for Mathematical Sciences (AIMS), Cameroon · Rochester Institute of Technology (RIT), US
Scaling Interpretable Transformers with Parity Bottleneck Layers
Jul 22, 2026Andrew Mack, Kraig Yuheng Tou, Mark Henry +2Feature Superposition Transformer Interpretability
Principles of Intelligence · Independent
SpikingMOT: A Spike-Driven Multi-Object Tracker
Jul 22, 2026Yiding Sun, Xiangyang Yang, Dongxu Zhang +7Activation Sparsity Spiking Neural Networks
1Xi’an Jiaotong University · 2Peking University · 3Guilin University of Technology
Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
Jul 19, 2026Aniket DeshpandeModel Compression Activation Sparsity
Department of Physics, University of Illinois Urbana-Champaign
Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation
Jul 16, 2026Qicheng Zhao, Qi Sun, Zheyu YanActivation Sparsity Diffusion Language Model Inference
Zhejiang University
Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Jul 9, 2026Bishmoy Paul, Youngmin Yi, Hoeseok YangLLM Pruning Activation Sparsity
Santa Clara University · Sogang University
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India
Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
Jul 2, 2026Rodrigo Mendoza-SmithSparse Recovery Activation Sparsity
Independent Researcher.
Accelerating Hierarchical Sparse Predictive Coding with Hybrid Amortized Inference
Jun 26, 2026Kazuhisa FujitaSparse Recovery Efficient Inference
Department of Clinical Engineering, Komatsu University, 10-10 Doihara-Machi, Komatsu, Ishikawa, Japan 923-0921
Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniActivation Sparsity Sparse Autoencoders
Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2
Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Jun 23, 2026Thomas Fel, Matthew Kowal, Mozes Jacobs +22Neural Representation Geometry Activation Sparsity
•Goodfire · aHarvard University · bStanford University +1
Effects of sparsity and superposition on loss in simple autoencoders
Jun 16, 2026Mriganka Basu Roy Chowdhury, Eric McLaughlin WeinerFeature Superposition Activation Sparsity
Department of Statistics, UC Berkeley · Department of Materials Science, UC Berkeley
Rational Sparse Autoencoder
Jun 12, 2026Naiyu Yin, Yue YuActivation Sparsity LLM Interpretability
Department of Mathematics · Lehigh University · Bethlehem, PA 18015
Decompose Sparsely Where You Should, Absorb Densely Where You Should No
Jun 12, 2026Ruixuan Deng, Zehao Jin, Zekun Wang +1Transformer Interpretability Representation Learning
School of Psychology Georgia Institute of Technology · College of Computing Georgia Institute of Technology · School of Interactive Computing Georgia Institute of Technology
Natively Unlearnable Large Language Models
Jun 11, 2026Gaurav R. Ghosal, Pratyush Maini, Aditi RaghunathanActivation Sparsity Machine Unlearning
Carnegie Mellon University · DatologyAI
Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs
Jun 2, 2026Ting LiuLLM Quantization On-Device Language Model Inference
SymbolicLight Research Foshan, Guangdong, China
Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations
May 27, 2026Bartosz Wieciech, Zmnako Awrahman, Marcin Czelej +2Activation Sparsity LLM Interpretability
Amazon Web Services
ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions
May 27, 2026Prathyush Poduval, Calvin Yeung, Neel Desai +1Transformer Interpretability Activation Sparsity
Department of Computer Science University of California, Irvine Irvine, CA 92697 · Georgia Institute of Technology Atlanta, GA 30332
RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models
May 26, 2026Xing Cong, Hanlin Tang, Kan Liu +3Diffusion Transformer Activation Sparsity
1Alibaba Group · 2Independent Researcher

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference

Mukund Agarwalla, Chih-Jen Lin

MBZUAI · National Taiwan University

Activation Sparsity LLM Inference Acceleration

Open paper AskPDF arXiv

Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not tightly control the realised sparsity, while TopK-based methods such as WINA enforce a fixed sparsity level but use the same sparsity budget for every token. Both also apply the same budget across transformer blocks, despite large differences in block sensitivity. We introduce TopK-Guided, a training-free method that addresses both limitations by combining bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation. Across Llama-2 and Llama-3 models, TopK-Guided consistently improves perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same sparsitydependent projection compute as WINA, with the largest gains at high sparsity. Ablations show that both components provide complementary improvements.

Momentum

Related topics

Latest papers 40