Activation Sparsity

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 40

All topics

Towards the Connection between Activation Sparsity and Flat Minima
May 25, 2026Ze Peng, Jian Zhang, Lei Qi +2Efficient Neural Network Inference Activation Sparsity
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Institute of Brain-Machine Interface, Nanjing University, Nanjing 210023, China · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China
Asymmetric Scaling Laws from Sparse Features
May 22, 2026John Sous, Michael WinerDouble Descent Activation Sparsity
Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA · Energy Sciences Institute, Yale University, West Haven, Connecticut 06516, USA · Institute for Advanced Study, Princeton, NJ 08540, USA +1
MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation
May 18, 2026Ronyu Zhang, Aosong Cheng, Gaole Dai +8Activation Sparsity Mixture of Experts
Nanjing University · The Hong Kong Polytechnic University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University +3
Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes
May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural Networks Activation Sparsity
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
May 5, 2026Jiaxi Li, Lu Yin, Li Shen +5Language Model Pretraining Activation Sparsity
University of Surrey · Sun Yat-sen University · Bytedance +2
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce
When Spike Sparsity Does Not Translate to Deployed Cost: VS-WNO on Jetson Orin Nano
Apr 18, 2026Jason Yoo, Shailesh Garg, Souvik Chakraborty +1Efficient Neural Network Inference Activation Sparsity
University of Illinois Urbana-Champaign Department of Nuclear, Plasma & Radiological Engineering Urbana, IL, USA · Indian Institute of Technology Delhi Department of Applied Mechanics New Delhi, India · National Center for Supercomputing Applications Urbana, IL, USA
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation Sparsity LLM Interpretability
University College London · MATS Research
Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
May 28, 2025Vadim Kurochkin, Yaroslav Aksenov, Daniil Laptev +2Representation Learning Activation Sparsity
T-Tech

May 25, 2026, cs.LGJ/K move · Enter open · S save

Towards the Connection between Activation Sparsity and Flat Minima

Ze Peng, Jian Zhang, Lei Qi, Yang Gao+1

State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Institute of Brain-Machine Interface, Nanjing University, Nanjing 210023, China · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China

Efficient Neural Network Inference Activation Sparsity Flat Minima Neural Network Training Dynamics Neural Network Pruning

Open paper AskPDF arXiv

The observation that activation sparsity emerges in MLP blocks of standardly trained Transformers offers an opportunity to drastically reduce computation costs without sacrificing performance. To theoretically explain this phenomenon, existing works have shown that activation sparsity does not result from the data properties or data fitting but from the implicit bias of the training process. However, these connections are obtained with strong assumptions, which cannot be applied to deep models standardly trained with a large number of steps. Different from these works, we find that the flatness of loss landscapes is also closely related to the MLP activation sparsity and can serve as a weaker and naturally emerging assumption standard deep networks. Specifically, we find that 1) the MLP activation sparsity equals a ratio between "augmented flatness" (a weighted sum of flatness measures) and the product of the input norm and activation gradient of the MLP. We empirically find that this ratio decreases during training, leading to sparse activations. 2) We also propose the notion of derivative sparsity, which reduces to activation sparsity under ReLU, but further enables pruning in the backward propagation and is more stable than activation sparsity. With the theoretical findings, we can further encourage activation sparsity by decreasing the numerator and increasing the denominator of the ratio using three methods. These plug-and-play modifications can effectively reduce the ratio and produce sparser activations. Experiments on ImageNet-1K and C4 demonstrate relative improvements of at least 36% on inference sparsity and at least 50% on training sparsity over vanilla Transformers, indicating further potential cost reduction in both inference and training

Momentum

Related topics

Latest papers 40