Sparse Neural Networks

Ask about this topic

Overview

Sparse neural networks study how to train, prune, or route models so that only selected weights, connections, features, or computations are active. Approaches include magnitude- and saliency-based pruning, dynamic sparse training that regrows connections, structured sparsity and feature-selection penalties, and sparsity-inducing Bayesian priors. The goal is to reduce memory and training or inference costs while retaining accuracy, across settings such as edge devices, federated learning, and large language models. Fewer parameters do not automatically mean faster execution or more interpretable models: hardware-aware layouts and kernels matter, while accuracy, robustness, and efficiency at extreme sparsity remain in tension.

Latest papers

CardsList

66 papers10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

  1. Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes

    Oct 23, 2026August Arnstad, Geir Olve Storvik, Filippo Pagani +1Bayesian Neural NetworksSparse Neural Networks

    Department of Mathematics, University of Oslo, NO · Department of Statistics, University of Warwick, UK

  2. Learning Computational Structure for Sample-Efficient Prediction

    Oct 20, 2026Hangyu Zhou, Constantine Dovrolis, Stephen MussmannData-Efficient LearningCompositional Representation Learning

    Georgia Institute of Technology · University of Cyprus

  3. Sparse Federated Echo State Networks for Time-Series Classification via Smooth-L1 Regularization

    Oct 15, 2026Gengcheng Lyu, Manuel Röder, Frank-Michael SchleifSparse Neural NetworksTime Series Classification

    Faculty of Computer Science and Business Information Systems Technical University of Applied Sciences Würzburg-Schweinfurt, Würzburg, Germany

  4. Optimizing Sparse Boolean Rule Networks using Mixed Integer Linear Programming

    Oct 14, 2026Thomas Pammer, Johannes Fürnkranz, Markus SinnlBoolean Function LearningInterpretable ML

    Institute for Application-Oriented Knowledge Processing (FAW), Johannes Kepler University Linz, Austria · LIT AI Lab, Johannes Kepler University Linz, Austria · Institute of Business Analytics & Technology Transformation/JKU Business School, Johannes Kepler University Linz, Austria

  5. When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

    Oct 12, 2026Mohammad Pasande, Mateusz Jaworski, Jeroen Rook +1Dynamic Sparse TrainingSparse Neural Networks

    University of Twente, Enschede, Netherlands · Paderborn University, Paderborn, Germany

  6. MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

    Oct 6, 2026Mingyuan Zhang, Yue Bai, Zhongruo Wang +5Expert RoutingSparse Neural Networks

    Department of Electrical and Computer Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University

  7. Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

    Oct 2, 2026Jafar Badour, Maurice van Keulen, Elena MocanuDynamic Sparse TrainingSparse Neural Networks

    University of Twente

  8. Do ResNets Route? Sparse Interaction Experts in Residual Networks

    Oct 2, 2026Liang Yan, Siying Chen, Kaijie Chen +3Residual LearningSparse Neural Networks

    Fudan University · University of Washington · Tongji University +2

  9. BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks

    Oct 1, 2026Mohammad Haroon Khawaja, Muhammad Haseeb, Mohammad Fatim Shoaib +1Neural Network PruningSparse Neural Networks

    Lahore University of Management Sciences Lahore, Pakistan

  10. Robust LassoNet: Enhancing Feature Selection in Neural Networks via Robust Loss Functions

    Sep 29, 2026Daniela De Canditiis, Italia De Feis, Paola StolfiRobust Loss FunctionsFeature Selection

    Istituto per le Applicazioni del Calcolo “Mauro Picone”, Consiglio Nazionale delle Ricerche, Via dei Taurini, 19, Roma, 00185, Italia · Istituto per le Applicazioni del Calcolo “Mauro Picone”, Consiglio Nazionale delle Ricerche, Via Pietro Castellino, 111, Napoli, 80131, Italia

  11. Sparsity-Adaptive Sharpness-Aware Minimization

    Sep 13, 2026Shiryu Ueno, Yoshikazu Hayashi, Kunihito KatoSparse Neural NetworksNeural Network Pruning

    Faculty of Engineering Gifu University Gifu, Japan

  12. NeuroFlex: Lossless Element-Level ANN-SNN Co-Execution for Efficient Sparse Inference

    Sep 12, 2026Varun Manjunath, Pranav Ramesh, Gopalakrishnan SrinivasanEfficient Neural Network InferenceAI Accelerator Inference

    Electrical Engineering Indian Institute of Technology Madras Chennai, India · Computer Science and Engineering Indian Institute of Technology Madras Chennai, India

  13. Sparse Competition during Training For the Emergence of Specialized Modules

    Aug 31, 2026Baptiste Rossigneux, Karim HarounSparse Neural NetworksNeural Network Training Dynamics

    Univ. Rennes · Inria, IRISA · University of Paris 8

  14. Selection, Representation, and Execution in Sparse Fourier Neural Operators

    Aug 30, 2026Abdul Qadir Ibrahim, Martin BurgerEfficient Neural Network InferenceFourier Neural Operator

    Deutsches Elektronen-Synchrotron DESY, Notkestraße 85, 22607 Hamburg, Germany

  15. Physics-Informed Foresight Pruning for Sparse PINN Solvers of Nonlinear PDEs

    Aug 26, 2026Ahmad Ishaque Karimi, Uvini Balasuriya Mudiyanselage, Kookjin LeeNeural PDE SolversNeural Network Pruning

    School of Computing and Augmented Intelligence Arizona State University Tempe, USA

  16. TabNSM: Neural Sparse Mixer for Tabular Regression

    Aug 18, 2026Ali Eslamian, Qiang ChengTabular RegressionTabular Representation Learning

    Department of Computer Science University of Kentucky Lexington, KY 40506 · Department of Computer Science Institute for Biomedical Informatics University of Kentucky Lexington, KY 40506

  17. Adaptive deep nonparametric regression from dependent data under covariate shift

    Jul 22, 2026William Kengne, Ehud Mossa OckegnaQuantile RegressionCovariate Shift

    Université Jean Monnet, ICJ UMR5208, CNRS, Ecole Centrale de Lyon, INSA Lyon, Universite Claude Bernard Lyon 1, 42000 Saint-Étienne, France

  18. SpikingMOT: A Spike-Driven Multi-Object Tracker

    Jul 22, 2026Yiding Sun, Xiangyang Yang, Dongxu Zhang +7Activation SparsitySpiking Neural Networks

    1Xi’an Jiaotong University · 2Peking University · 3Guilin University of Technology

  19. Event-based Neural Decoding for Neuroprosthetic Motor Control

    Jul 13, 2026Khaleelulla Khan Nazeer, Sirine Arfa, Matthias Jobst +2Efficient Neural Network InferenceBrain-Computer Interfaces

    Chair of Highly-Parallel VLSI-Systems and Neuromorphic Circuits, Technische Universität Dresden, Germany · Centre for Tactile Internet with Human-in-the-Loop (CeTI), Dresden, Germany

  20. Individual Parameters in Weight-Sparse Transformers Appear Interpretable

    Jul 3, 2026Arnau Marin-Llobet, Stefan HeimersheimTransformer InterpretabilityTransformer

    School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138

  21. End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

    Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM InferenceStructured Sparsity

    University of North Carolina at Chapel Hill · Meta AI

  22. Local Pheromone Network: Sparse Local Learning with Multi-Scale Synaptic Trails, Consolidation, and Replay

    Jun 22, 2026Xingcheng Fu, Xianjun Chen, Zhihao LiHebbian LearningContinual Learning

  23. Distilling a Modular Reservoir Through a Genomic Bottleneck

    Jun 20, 2026Mani Hamidi, Sina Khajehabdollahi, Charley M. Wu +1Reservoir ComputingSparse Neural Networks

    Department of Computer Science, University of Tübingen, Tübingen, Germany · Flowers Team Inria, Bordeaux, France · Max Planck Institute for Biological Cybernetics, Tübingen, Germany +3

  24. Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

    Jun 18, 2026Dat H. Do, Rushi Shah, Duc V. Le +1Sparse Neural NetworksCompositional Representation Learning

    National University of Singapore1 · University of Twente2

  25. Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

    Jun 10, 2026Romana Qureshi, Hafida Benhidour, Said Kerrache +1Sparse Neural NetworksNeural Network Pruning

    Department of Computer Science, King Saud University, Riyadh, Saudi Arabia

  26. DAGGER: Gradient-Free Construction of Transiently Amplifying Networks under Hard Connectivity Constraints

    May 31, 2026James C. FergusonDynamical SystemsGradient-Free Optimization

    The African Institute for Mathematical Sciences (Muizenberg, South Africa) · the Institute of Science and Technology Austria (Klosterneuburg, Austria)

  27. HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces

    May 31, 2026Nasib Ullah, Jinbin Zhang, Jean Lucien Randrianantenaina +2Multi-Label ClassificationStructured Sparsity

    Department of Computer Science, Aalto University, Espoo, Finland · Department of Computer Science, University of Bath, Bath, UK. · IST Austria.

  28. BIRDNet: Mining and Encoding Boolean Implication Knowledge Graphs as Interpretable Deep Neural Networks

    May 27, 2026Tirtharaj DashNeural Network InterpretabilityNeuro-Symbolic Reasoning

    BITS Pilani, K K Birla Goa Campus Zuarinagar, Goa 403726, India

  29. PrunePath: Towards Highly Structured Sparse Language Models

    May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts PruningLLM Pruning

    Department of Applied Mathematics, The Hong Kong Polytechnic University

  30. Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

    May 27, 2026Bartosz Wieciech, Zmnako Awrahman, Marcin Czelej +2Activation SparsityLLM Interpretability

    Amazon Web Services