Dynamic Sparse Training

Dynamic sparse training (DST) aims to improve the efficiency of deep learning models by training sparse neural networks from scratch, dynamically adjusting the network's connectivity during training to achieve high sparsity levels while maintaining accuracy comparable to dense models. Current research focuses on optimizing pruning and growing criteria, exploring different sparsity structures (e.g., unstructured, channel-level, N:M), and adapting DST to various architectures (including transformers and GANs) and learning paradigms (like federated learning and continual learning). This approach offers significant potential for reducing computational costs and memory requirements in various applications, from natural language processing and computer vision to resource-constrained environments like edge devices and embedded systems.

Papers

November 5, 2024

Navigating Extremes: Dynamic Sparsity in Large Output Space
Nasib Ullah, Erik Schultheis, Mike Lasby, Yani Ioannou, Rohit Babbar
Dynamic Sparse Training Extreme Value Output Space Dynamic Sparsity Sparse Matrix Multiplication Sparse Classification

October 3, 2024

Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness
Boqian Wu, Qiao Xiao, Shunxin Wang, Nicola Strisciuglio, Mykola Pechenizkiy, Maurice van Keulen, Decebal Constantin Mocanu, Elena Mocanu
Dynamic Sparse Training Neural Network Robustness Dense Training Semantic Corruption

September 28, 2024

Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training
Pihe Hu, Shaolong Li, Zhuoran Li, Ling Pan, Longbo Huang
Multi Agent Multi Agent Reinforcement Dynamic Sparse Training Agent Training

August 21, 2024

Mixed Sparsity Training: Achieving 4$\times$ FLOP Reduction for Transformer Pretraining
Pihe Hu, Shaolong Li, Longbo Huang
Large Language Model Sparse Attention Transformer Based Large Language Model Dynamic Sparse Training

August 8, 2024

Unveiling the Power of Sparse Neural Networks for Feature Selection
Zahra Atashgahi, Tennison Liu, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu, Mihaela van der Schaar
Feature Selection Real Power Sparse Neural Network Dense Network Dynamic Sparse Training

June 3, 2024

Sparsity-Accelerated Training for Large Language Models
Da Ma, Lu Chen, Pengyu Wang, Hongshen Xu, Hanqi Li, Liangtai Sun, Su Zhu, Shuai Fan, Kai Yu
Large Language Model Natural Language Processing Supervised Fine Tuning Dynamic Sparse Training Neuron Importance

May 29, 2024

UniPTS: A Unified Framework for Proficient Post-Training Sparsity
Jingjing Xie, Yuxin Zhang, Mingbao Lin, Zhihang Lin, Liujuan Cao, Rongrong Ji
Unified Framework Sparse Network Dynamic Sparse Training Network Sparsity Post Training Sparsity

May 24, 2024

Sparse maximal update parameterization: A holistic approach to sparse training dynamics
Nolan Dey, Shane Bergsma, Joel Hestness
Many Sparse Sparse Model Sparse Neural Network Holistic Approach Dense Network Dynamic Sparse Training

March 5, 2024

DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecasting
Hao Wu, Haomin Wen, Guibin Zhang, Yutong Xia, Kai Wang, Yuxuan Liang, Yu Zheng, Kun Wang
Spatio Temporal Forecasting Sensor Placement Dynamic Sparse Training Sensor Trace

February 23, 2024

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning
Zhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li, Jianling Sun
Pre Trained Language Model Parameter Efficient Fine Tuning Mixture of Expert Parameter Efficient Transfer Learning Task Specific Knowledge Dynamic Sparse Training Task Correlation Multi Task Transfer

February 9, 2024

Learn To be Efficient: Build Structured Sparsity in Large Language Models
Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z. Morley Mao, Beidi Chen, Fan Lai, Atul Prakash
High Efficiency Inference Cost Structured Sparsity Activation Sparsity Dynamic Sparse Training

February 5, 2024

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
Shuyao Wang, Yongduo Sui, Jiancan Wu, Zhi Zheng, Hui Xiong
Deep Learning Recommendation Model New Paradigm Recommendation Quality Dynamic Sparse Training

October 13, 2023

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, Rongrong Ji
Large Language Model Training Data Training Free Dynamic Sparse Training Dynamic Sparsity

August 28, 2023

Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates
Murat Onur Yildirim, Elif Ceren Gok Yildirim, Ghada Sokar, Decebal Constantin Mocanu, Joaquin Vanschoren
Continual LEArning Practical Algorithm Continual Learning Sparse Network Dynamic Sparse Training Model Update

June 21, 2023

Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training
Aleksandra I. Nowak, Bram Grooten, Decebal Constantin Mocanu, Jacek Tabor
Edge Pruning Balancing Weight Sparse Connectivity Dynamic Sparse Training Magnitude Based Pruning

June 2, 2023

Group channel pruning and spatial attention distilling for object detection
Yun Chu, Pu Li, Yong Bai, Zhuhua Hu, Yongqing Chen, Jiafeng Lu
Object Detection Model Compression Channel Pruning Dynamic Sparse Training Attention Distillation

May 30, 2023

Dynamic Sparsity Is Channel-Level Sparsity Learner
Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu
Dynamic Sparse Training Dynamic Sparsity Channel Sparsity

May 3, 2023

Dynamic Sparse Training with Structured Sparsity
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, Yani Ioannou
Structured Sparsity Sparse Training Dynamic Sparse Training Unstructured Sparsity

February 28, 2023

Balanced Training for Sparse GANs
Yite Wang, Jing Wu, Naira Hovakimyan, Ruoyu Sun
Generative Adversarial Network GAN Training Dynamic Sparse Training

February 13, 2023

Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning
Bram Grooten, Ghada Sokar, Shibhansh Dohare, Elena Mocanu, Matthew E. Taylor, Mykola Pechenizkiy, Decebal Constantin Mocanu
Reinforcement Learning Deep Reinforcement Learning Noisy Environment Dynamic Sparse Training Task Specific Feature Deep Reinforcement Learning Algorithm