Activation Sparsity

Ask about this topic

Overview

Activation sparsity studies when and how neural networks can leave units inactive for particular inputs, potentially saving computation and memory or yielding more interpretable features. Common approaches include Top-k and magnitude thresholding, structured N:M patterns, input-conditioned gates, and sensitivity-aware allocation of sparsity across tokens and layers; sparse autoencoders use related constraints to analyze model representations. Applications range from large language model and diffusion inference to edge vision and spiking networks. The central tension is that higher sparsity can damage accuracy or feature reliability, and theoretical operation savings translate into real speed or energy gains only when hardware and software efficiently skip inactive work.

Latest papers

CardsList

56 papers4 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

  1. BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation CompressionMemory Compression

    University of Cambridge · University of Oxford

  2. Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

    Oct 1, 2026JuneHyung Kim, Sankeerth Durvasula, Nandita VijaykumarLLM InferenceLLM Inference Acceleration

    LG Electronics · University of Toronto

  3. TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference

    Oct 1, 2026Mukund Agarwalla, Chih-Jen LinActivation SparsityLLM Inference Acceleration

    MBZUAI · National Taiwan University

  4. Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability

    Sep 29, 2026Zhenting Huang, Bo Jiang, Junnan Liu +2Activation SparsityLLM Interpretability

    Beihang University · Monash University · Zhongguancun Laboratory

  5. Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

    Sep 27, 2026Jungseob Lee, Seungyoon Lee, Seongtae Hong +2Language Model DecodingActivation Sparsity

    Korea University · Yonsei University Mirae Campus

  6. Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

    Sep 25, 2026Maria Lomeli, Antoine Groudiev, Matthijs Douze +7Efficient Neural Network InferenceTransformer FFNs

    Meta FAIR · École Normale Supérieure – PSL · École Normale Supérieure Paris-Saclay

  7. SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking

    Sep 21, 2026Pengzhi Zhong, Jiwei Mo, Haolun Li +4Activation SparsityVisual Object Tracking

    College of Computer Science and Engineering, Guilin University of Technology, Guilin 541004, China · School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China

  8. Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks

    Sep 8, 2026Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2Neural Network GeneralizationReLU Neural Networks

    University of New South Wales · University of Texas at Austin · University of Sydney

  9. SPARK: Input-Conditioned Sparse Activation Modulation for Frozen DiT-based Super-Resolution

    Sep 3, 2026Federico Putamorsi, Leonardo Zini, Marcella Cornia +1Diffusion Models for Image RestorationDiffusion Transformer

    University of Modena and Reggio Emilia, Italy

  10. Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware

    Aug 31, 2026Simon Richter, Ruhai Lin, Jason Yik +4Activation SparsityLLM Inference Acceleration

    Department of Electrical and Computer Engineering, Aarhus University, Aarhus, Denmark · Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, USA · School of Engineering and Applied Sciences, Harvard University, Cambridge, USA +1

  11. Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Neuromorphic Virtual Sensing

    Aug 25, 2026William Howes, Farid Ahmed, Syed Bahauddin AlamActivation SparsityNeuromorphic Computing

    Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign, Urbana, IL, USA · National Center for Supercomputing Applications, Urbana, IL, USA

  12. A Privacy Study of Sparse Collaborative Inference

    Aug 17, 2026Maximilian Andreas Hoefler, Karsten Mueller, Wojciech SamekPrivacy-Preserving MLActivation Sparsity

    Fraunhofer Heinrich Hertz Institute, Germany · Technical University of Berlin, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD)

  13. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

    Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Activation SparsityStructured Sparsity

    Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications

  14. The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy

    Jul 29, 2026Zeyu WangEfficient Neural Network InferenceSpiking Transformers

    Georgia Institute of Technology

  15. At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

    Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Efficient Neural Network InferenceTransformer Inference

    *IIS, ETH Zürich, Switzerland · ‡Computing Systems Lab, Huawei, Switzerland · †DEI, Universit`a di Bologna, Italy

  16. Neural Feature Governance: Extending Atom Prevalence

    Jul 23, 2026Idris Karel Seunda Ekwe, Patrick Tenga Shako, Ernest Parfait FokouéBayesian Neural NetworksUncertainty Quantification

    African Institute for Mathematical Sciences (AIMS), Cameroon · Rochester Institute of Technology (RIT), US

  17. Scaling Interpretable Transformers with Parity Bottleneck Layers

    Jul 22, 2026Andrew Mack, Kraig Yuheng Tou, Mark Henry +2Feature SuperpositionTransformer Interpretability

    Principles of Intelligence · Independent

  18. SpikingMOT: A Spike-Driven Multi-Object Tracker

    Jul 22, 2026Yiding Sun, Xiangyang Yang, Dongxu Zhang +7Activation SparsitySpiking Neural Networks

    1Xi’an Jiaotong University · 2Peking University · 3Guilin University of Technology

  19. Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?

    Jul 19, 2026Aniket DeshpandeModel CompressionActivation Sparsity

    Department of Physics, University of Illinois Urbana-Champaign

  20. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    Jul 16, 2026Qicheng Zhao, Qi Sun, Zheyu YanActivation SparsityDiffusion Language Model Inference

    Zhejiang University

  21. Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models

    Jul 9, 2026Bishmoy Paul, Youngmin Yi, Hoeseok YangLLM PruningActivation Sparsity

    Santa Clara University · Sogang University

  22. It Takes a MAESTRO To Prune Bad Experts

    Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts PruningLLM Pruning

    Indian Institute of Technology Delhi, India · NVIDIA, India

  23. Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability

    Jul 2, 2026Rodrigo Mendoza-SmithSparse RecoveryActivation Sparsity

    Independent Researcher.

  24. Accelerating Hierarchical Sparse Predictive Coding with Hybrid Amortized Inference

    Jun 26, 2026Kazuhisa FujitaSparse RecoveryEfficient Inference

    Department of Clinical Engineering, Komatsu University, 10-10 Doihara-Machi, Komatsu, Ishikawa, Japan 923-0921

  25. Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

    Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniActivation SparsitySparse Autoencoders

    Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2

  26. Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds

    Jun 23, 2026Thomas Fel, Matthew Kowal, Mozes Jacobs +22Neural Representation GeometryActivation Sparsity

    •Goodfire · aHarvard University · bStanford University +1

  27. Effects of sparsity and superposition on loss in simple autoencoders

    Jun 16, 2026Mriganka Basu Roy Chowdhury, Eric McLaughlin WeinerFeature SuperpositionActivation Sparsity

    Department of Statistics, UC Berkeley · Department of Materials Science, UC Berkeley

  28. Rational Sparse Autoencoder

    Jun 12, 2026Naiyu Yin, Yue YuActivation SparsityLLM Interpretability

    Department of Mathematics · Lehigh University · Bethlehem, PA 18015

  29. Decompose Sparsely Where You Should, Absorb Densely Where You Should No

    Jun 12, 2026Ruixuan Deng, Zehao Jin, Zekun Wang +1Transformer InterpretabilityRepresentation Learning

    School of Psychology Georgia Institute of Technology · College of Computing Georgia Institute of Technology · School of Interactive Computing Georgia Institute of Technology

  30. Natively Unlearnable Large Language Models

    Jun 11, 2026Gaurav R. Ghosal, Pratyush Maini, Aditi RaghunathanActivation SparsityMachine Unlearning

    Carnegie Mellon University · DatologyAI