Sparse Network Training

Sparse network training aims to develop efficient deep learning models by reducing the number of parameters while maintaining or improving performance. Current research focuses on methods that encourage sparsity during training, including techniques that leverage inter-layer feature similarity and those that generate and combine multiple sparse subnetworks. This area is significant because it promises to reduce computational costs, energy consumption, and memory requirements for deep learning, making it more accessible and environmentally friendly while potentially improving model generalization.

Papers

July 14, 2023

Learning Sparse Neural Networks with Identity Layers
Mingjian Ni, Guangyao Chen, Xiawu Zheng, Peixi Peng, Li Yuan, Yonghong Tian
Sparse Network Sparse Neural Network Network Sparsity Sparsity Regularization Sparse Network Training

March 30, 2023

BOLT: An Automated Deep Learning Framework for Training and Deploying Large-Scale Search and Recommendation Models on Commodity CPU Hardware
Nicholas Meisburger, Vihan Lakshman, Benito Geordie, Joshua Engels, David Torres Ramos, Pratik Pranav, Benjamin Coleman, Benjamin Meisburger, Shubh Gupta, Yashwanth Adunukota, Tharun Medini, Anshumali Shrivastava
Deep Learning Recommendation Model Large Scale Search Neural Network Training Sparse Network Training

February 6, 2023

Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers
Shiwei Liu, Zhangyang Wang
Critical Lesson Sparse Neural Network Sparse Training Dynamic Sparsity Sparse to Sparse Training Sparse Network Training Post Training Sparsity

May 30, 2022

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Decebal Constantin Mocanu, Shiwei Liu
Performance Improvement Sparse Neural Network Sparse Training Sparse Subnetworks Dynamic Sparse Training Robust Ticket Sparse Network Training

April 4, 2022

APP: Anytime Progressive Pruning
Diganta Misra, Bharat Runwal, Tianlong Chen, Zhangyang Wang, Irina Rish
Long Sequence Large Scale Learning Progressive Pruning Sparse Network Training Target Sparsity

March 8, 2022

Dual Lottery Ticket Hypothesis
Yue Bai, Huan Wang, Zhiqiang Tao, Kunpeng Li, Yun Fu
Sparse Network Sparse Neural Network Lottery Ticket Hypothesis Sparse Network Training