Dynamic Sparse Training

Follow this topic Ask about this topic

Latest papers

10 papers4 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training
Oct 12, 2026Mohammad Pasande, Mateusz Jaworski, Jeroen Rook +1Dynamic Sparse Training Sparse Neural Networks
University of Twente, Enschede, Netherlands · Paderborn University, Paderborn, Germany
BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels
Oct 12, 2026Yueyuan Sui, Payal Mohapatra, Qi Zhu +1Dynamic Sparse Training Neural Network Pruning
Northwestern University, USA · Arm Inc., USA
DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
Oct 8, 2026Bhushan DeshpandeDynamic Sparse Training Neural Network Pruning
Independent Researcher, Pune, India
Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU
Oct 2, 2026Jafar Badour, Maurice van Keulen, Elena MocanuDynamic Sparse Training Sparse Neural Networks
University of Twente
HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
May 31, 2026Nasib Ullah, Jinbin Zhang, Jean Lucien Randrianantenaina +2Multi-Label Classification Structured Sparsity
Department of Computer Science, Aalto University, Espoo, Finland · Department of Computer Science, University of Bath, Bath, UK. · IST Austria.
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Language Model Pretraining Efficient Language Model Training
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
May 26, 2026Mohammed Adnan, Rohan Jain, Tom Jacobs +4Neural Network Optimization Dynamic Sparse Training
University of Calgary · 2Vector Institute · 3CISPA Helmholtz +2
Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria
Apr 12, 2026Nikodem TomczakDynamic Sparse Training Sparse Neural Networks
Thulge Labs, Singapore
Growing Networks with Autonomous Pruning
Mar 20, 2026Charles De Lambilly, Stefan DuffnerDynamic Sparse Training Neural Network Growth
INSA Lyon, CNRS, Ecole Centrale de Lyon, Universite Claude Bernard Lyon 1, Université Lumière Lyon 2, LIRIS, UMR5205, , Villeurbanne, 69621, France
Robustness in sparse artificial neural networks trained with adaptive topology
Feb 25, 2026Bendegúz Sulyok, Gergely Palla, Filippo Radicchi +1Dynamic Sparse Training Sparse Neural Networks
Dept. of Biological Physics, Eötvös Loránd University, Budapest, Hungary · Semmelweis University, Faculty of Health and Public Administration, Health Services Management Training Centre, Budapest, Hungary · Center for Complex Networks and Systems Research Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington, USA

Oct 12, 2026J/K move · Enter open · S save

When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Mohammad Pasande, Mateusz Jaworski, Jeroen Rook, Elena Mocanu

University of Twente, Enschede, Netherlands · Paderborn University, Paderborn, Germany

Dynamic Sparse Training Sparse Neural Networks Neural Network Pruning Neural Network Training Dynamics

Open paper AskPDF arXiv

Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies. At extreme sparsity (

\ge 99\%

), this dynamic tends to destroy input-output information pathways, and performance with standard magnitude/random updates degrades sharply. We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory. Guided by this reachability-based structure, we bias pruning toward non-functional edges and regrowth toward functionally consistent candidates. We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST. On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to

+14.4

points at

99.9\%

sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at

99.99\%

sparsity while unguided random regrowth degrades sharply. These results cast extreme-sparse training as a connectivity-preserving dynamical process. Our implementation is publicly available at https://github.com/mzjaworski/ea_dst.