Random Pruning

Random pruning, a technique for reducing the complexity of neural networks by randomly removing parameters or data points, is a surprisingly effective method for improving model efficiency and sometimes even accuracy. Current research focuses on understanding why random pruning works so well, comparing it to more sophisticated pruning methods across various architectures (including convolutional and generative language models), and optimizing the sparsity levels for different network types and training regimes. This research is significant because it offers a computationally inexpensive approach to model compression, potentially leading to faster training and inference times, reduced memory requirements, and improved generalization in deep learning applications.

Papers

October 4, 2023

Enhancing Accuracy in Deep Learning Using Random Matrix Theory
Leonid Berlyand, Etienne Sandier, Yitzchak Shmalo, Lei Zhang
Neural Network Deep Learning Random Matrix Enhancing Accuracy Deep Neural Network Parameter Layer Pruning Random Pruning

March 26, 2023

February 14, 2023

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms
Fadhel Ayed, Soufiane Hayou
Fundamental Limitation Score Based Data Pruning Neural Scaling Law Random Pruning

February 7, 2023

What Matters In The Structured Pruning of Generative Language Models?
Michael Santacroce, Zixin Wen, Yelong Shen, Yuanzhi Li
Generative Language Model Structured Pruning Random Pruning Structured Pruning Method

October 5, 2022

Why Random Pruning Is All We Need to Start Sparse
Advait Gadhikar, Sohom Mukherjee, Rebekka Burkholz
Many Sparse Sparse Network Random Network Random Pruning

May 11, 2022

Revisiting Random Channel Pruning for Neural Network Compression
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, Luc Van Gool
Neural Network Compression Channel Pruning Channel Selection Random Pruning Channel Importance

March 27, 2022

On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks
Hongru Yang, Zhangyang Wang
Neural Network Neural Tangent Kernel Random Participation Fully Connected Neural Network Random Pruning

February 23, 2022

Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks
Tim Whitaker, Darrell Whitley
Ensemble Learning Diverse Ensemble Edge Pruning Sparse Subnetworks Random Pruning Low Cost Ensemble

February 5, 2022

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy
Customer Service Sparse Network Sparse Training Naive Ensemble Random Pruning Layerwise Sparsity

November 24, 2021

Accelerating Deep Learning with Dynamic Data Pruning
Ravi S Raju, Kyle Daruwalla, Mikko Lipasti
Deep Learning Score Based Data Pruning Random Pruning

Random Pruning

Papers

Enhancing Accuracy in Deep Learning Using Random Matrix Theory

Exploring the Performance of Pruning Methods in Neural Networks: An Empirical Study of the Lottery Ticket Hypothesis

Does "Deep Learning on a Data Diet" reproduce? Overall yes, but GraNd at Initialization does not

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

What Matters In The Structured Pruning of Generative Language Models?

Why Random Pruning Is All We Need to Start Sparse

Revisiting Random Channel Pruning for Neural Network Compression

On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks

Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

Accelerating Deep Learning with Dynamic Data Pruning