Sparse Training

Sparse training aims to reduce the computational cost and memory footprint of deep neural networks by training models with significantly fewer parameters, while maintaining or even improving accuracy. Current research focuses on developing efficient algorithms for creating and training sparse models, including methods for dynamic sparsity adjustment, improved initialization techniques, and hardware-accelerated computations, often applied to transformer and convolutional neural networks. These advancements are significant because they enable the deployment of large-scale models on resource-constrained devices and reduce the environmental impact of training, impacting both scientific research and practical applications in various fields.

Papers

June 26, 2022

Training Your Sparse Neural Network Better with Any Mask
Ajay Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, Zhangyang Wang
Sparse Neural Network Sparse Training Mask Frozen DETR Sparse Mask

June 17, 2022

The State of Sparse Training in Deep Reinforcement Learning
Laura Graesser, Utku Evci, Erich Elsen, Pablo Samuel Castro
Deep Reinforcement Learning Internal State Sparse Network Sparse Neural Network Sparse Training

June 9, 2022

DiSparse: Disentangled Sparsification for Multitask Model Compression
Xinglong Sun, Ali Hassani, Zhangyang Wang, Gao Huang, Humphrey Shi
Model Compression Multitask Learning Random Sparsification Sparse Training Multitask Model

May 30, 2022

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Decebal Constantin Mocanu, Shiwei Liu
Performance Improvement Sparse Neural Network Sparse Training Sparse Subnetworks Dynamic Sparse Training Robust Ticket Sparse Network Training

May 23, 2022

Parameter-Efficient Sparsity for Large Language Models Fine-Tuning
Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, Junjie Bai
Large Scale Language Model Sparse Training Parameter Efficient Sparsity Importance Weight

February 5, 2022

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy
Customer Service Sparse Network Sparse Training Naive Ensemble Random Pruning Layerwise Sparsity

November 30, 2021

Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher Ré
Neural Network Model Sparse Model Sparse Training Sparse Matrix Sparse Mask Butterfly Matrix Monarch Butterfly

November 10, 2021

Efficient Neural Network Training via Forward and Backward Propagation Sparsification
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, Tong Zhang
Neural Network Sparse Training Sparse Structure Backward Propagation Sparsity Constraint