Sparse Neural Networks

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 32

All topics

MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
Oct 6, 2026Mingyuan Zhang, Yue Bai, Zhongruo Wang +5Expert Routing Sparse Neural Networks
Department of Electrical and Computer Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University
Robust LassoNet: Enhancing Feature Selection in Neural Networks via Robust Loss Functions
Sep 29, 2026Daniela De Canditiis, Italia De Feis, Paola StolfiRobust Loss Functions Feature Selection
Istituto per le Applicazioni del Calcolo “Mauro Picone”, Consiglio Nazionale delle Ricerche, Via dei Taurini, 19, Roma, 00185, Italia · Istituto per le Applicazioni del Calcolo “Mauro Picone”, Consiglio Nazionale delle Ricerche, Via Pietro Castellino, 111, Napoli, 80131, Italia
NeuroFlex: Lossless Element-Level ANN-SNN Co-Execution for Efficient Sparse Inference
Sep 12, 2026Varun Manjunath, Pranav Ramesh, Gopalakrishnan SrinivasanEfficient Neural Network Inference AI Accelerator Inference
Electrical Engineering Indian Institute of Technology Madras Chennai, India · Computer Science and Engineering Indian Institute of Technology Madras Chennai, India
Sparse Competition during Training For the Emergence of Specialized Modules
Aug 31, 2026Baptiste Rossigneux, Karim HarounSparse Neural Networks Neural Network Training Dynamics
Univ. Rennes · Inria, IRISA · University of Paris 8
Selection, Representation, and Execution in Sparse Fourier Neural Operators
Aug 30, 2026Abdul Qadir Ibrahim, Martin BurgerEfficient Neural Network Inference Fourier Neural Operator
Deutsches Elektronen-Synchrotron DESY, Notkestraße 85, 22607 Hamburg, Germany
Adaptive deep nonparametric regression from dependent data under covariate shift
Jul 22, 2026William Kengne, Ehud Mossa OckegnaQuantile Regression Covariate Shift
Université Jean Monnet, ICJ UMR5208, CNRS, Ecole Centrale de Lyon, INSA Lyon, Universite Claude Bernard Lyon 1, 42000 Saint-Étienne, France
SpikingMOT: A Spike-Driven Multi-Object Tracker
Jul 22, 2026Yiding Sun, Xiangyang Yang, Dongxu Zhang +7Activation Sparsity Spiking Neural Networks
1Xi’an Jiaotong University · 2Peking University · 3Guilin University of Technology
Event-based Neural Decoding for Neuroprosthetic Motor Control
Jul 13, 2026Khaleelulla Khan Nazeer, Sirine Arfa, Matthias Jobst +2Efficient Neural Network Inference Brain-Computer Interfaces
Chair of Highly-Parallel VLSI-Systems and Neuromorphic Circuits, Technische Universität Dresden, Germany · Centre for Tactile Internet with Human-in-the-Loop (CeTI), Dresden, Germany
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
Jul 3, 2026Arnau Marin-Llobet, Stefan HeimersheimTransformer Interpretability Transformer
School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138
End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM Inference Structured Sparsity
University of North Carolina at Chapel Hill · Meta AI
Local Pheromone Network: Sparse Local Learning with Multi-Scale Synaptic Trails, Consolidation, and Replay
Jun 22, 2026Xingcheng Fu, Xianjun Chen, Zhihao LiHebbian Learning Continual Learning
Distilling a Modular Reservoir Through a Genomic Bottleneck
Jun 20, 2026Mani Hamidi, Sina Khajehabdollahi, Charley M. Wu +1Reservoir Computing Sparse Neural Networks
Department of Computer Science, University of Tübingen, Tübingen, Germany · Flowers Team Inria, Bordeaux, France · Max Planck Institute for Biological Cybernetics, Tübingen, Germany +3
Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds
Jun 18, 2026Dat H. Do, Rushi Shah, Duc V. Le +1Sparse Neural Networks Compositional Representation Learning
National University of Singapore1 · University of Twente2
Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
Jun 10, 2026Romana Qureshi, Hafida Benhidour, Said Kerrache +1Sparse Neural Networks Neural Network Pruning
Department of Computer Science, King Saud University, Riyadh, Saudi Arabia
DAGGER: Gradient-Free Construction of Transiently Amplifying Networks under Hard Connectivity Constraints
May 31, 2026James C. FergusonDynamical Systems Sparse Neural Networks
The African Institute for Mathematical Sciences (Muizenberg, South Africa) · the Institute of Science and Technology Austria (Klosterneuburg, Austria)
HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
May 31, 2026Nasib Ullah, Jinbin Zhang, Jean Lucien Randrianantenaina +2Multi-Label Classification Structured Sparsity
Department of Computer Science, Aalto University, Espoo, Finland · Department of Computer Science, University of Bath, Bath, UK. · IST Austria.
BIRDNet: Mining and Encoding Boolean Implication Knowledge Graphs as Interpretable Deep Neural Networks
May 27, 2026Tirtharaj DashNeural Network Interpretability Neuro-Symbolic Reasoning
BITS Pilani, K K Birla Goa Campus Zuarinagar, Goa 403726, India
PrunePath: Towards Highly Structured Sparse Language Models
May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts Pruning LLM Pruning
Department of Applied Mathematics, The Hong Kong Polytechnic University
Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations
May 27, 2026Bartosz Wieciech, Zmnako Awrahman, Marcin Czelej +2Activation Sparsity LLM Interpretability
Amazon Web Services
Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
May 23, 2026Leonid Berlyand, Theo Bourdais, Houman Owhadi +1Structured Pruning Sparse Neural Networks
Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Encoder Spike Sparsity
May 20, 2026Ting LiuLanguage Model Pretraining Spiking Neural Networks
SymbolicLight Research Foshan, Guangdong, China
Stimulus symmetries can confound representational similarity analyses
May 20, 2026Farhad Pashakhanloo, Jacob A. Zavatone-VethRepresentational Similarity Analysis Representation Learning
Center for Brain Science and 2Society of Fellows Harvard University, Cambridge, MA, USA
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzDeep Learning Optimization Transformer
CISPA Helmholtz Center for Information Security
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
May 17, 2026Hoai-Chau Tran, Chi H. Nguyen, Duy M. H. Nguyen +3Efficient Transformer Inference Image Segmentation
University of Illinois at Urbana-Champaign · VinUni-Illinois Smart Health Center, VinUniversity · College of Engineering & Computer Science, VinUniversity +3
Compositional Sparsity as an Inductive Bias for Neural Architecture Design
May 14, 2026Hongyu Lin, Antonio Briola, Yuanrong Wang +1Hierarchical Representation Learning Structured Sparsity
Department of Computer Science, University College London, London, United Kingdom
Posterior Contraction Rates for Sparse Kolmogorov-Arnold Networks in Anisotropic Besov Spaces
May 12, 2026Jeunghun Oh, Kyeongwon Lee, Jaeyong Lee +1Bayesian Neural Networks Kolmogorov-Arnold Networks
Department of Statistics, Seoul National University · Department of Statistics, Inha University · Department of Mathematics, University of Maryland
Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features
May 10, 2026Guangqi Li, Yongxin LiSparse Neural Networks Neural Network Compression
Zaozhuang University, Zaozhuang, Shandong, China
Adaptive Regularization for Sparsity Control in Bregman-Based Optimizers
May 8, 2026Ahmad Aloradi, Tim Roith, Emanuël A. P. Habets +1Neural Network Optimization Sparse Neural Networks
Department of Data Science, FAU Erlangen-Nürnberg, Erlangen, Germany · 2International Audiolabs, FAU Erlangen-Nürnberg, Erlangen, Germany · School of Computation, Information and Technology, Technical University of Munich +1
S-AI-Recursive: A Bio-Inspired and Temporal Sparse AI Architecture for Iterative, Introspective, and Energy-Frugal Reasoning
May 5, 2026Said SlaouiEnergy-Efficient ML Sparse Neural Networks
Mohammed V University, Rabat, Morocco
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang+4

Department of Electrical and Computer Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University

Expert Routing Sparse Neural Networks Mixture-of-Experts Models

Open paper AskPDF arXiv

Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation. Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts. We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN. Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine. The router and mask scores are optimized jointly, while the underlying FFN weight values remain unchanged. This formulation supports neuron-structured, semi-structured, and unstructured experts within the same routing architecture. Our main configuration uses four 2:4 experts with top-2 routing, where two half-dense expert passes have the nominal FFN arithmetic of one dense pass, without requiring independent expert weight matrices. On five vision-language benchmarks with Qwen and Gemma backbones, this configuration achieves the highest performance among the compared baselines. Comparisons across mask granularities, routing interventions, and compute-matched controls distinguish the effects of learned connectivity from expert activation count. These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts.

Momentum

Related topics

Latest papers 32