Sparse Neural Network

Sparse neural networks (SNNs) aim to improve the efficiency and interpretability of deep learning models by reducing the number of parameters while maintaining or even exceeding the performance of their dense counterparts. Current research focuses on developing novel pruning algorithms, exploring the interplay between data and model architecture in achieving sparsity, and investigating the impact of sparsity on training dynamics and generalization. This area is significant because SNNs offer the potential for reduced computational costs, improved energy efficiency, and enhanced model interpretability, leading to wider deployment of deep learning in resource-constrained environments and applications requiring explainability.

Papers

October 3, 2024

RelChaNet: Neural Network Feature Selection using Relative Change Scores
Felix Zimmer
Feature Selection MNIST Dataset Sparse Neural Network Dense Neural Network Neuron Pruning

September 13, 2024

Are Sparse Neural Networks Better Hard Sample Learners?
Qiao Xiao, Boqian Wu, Lu Yin, Christopher Neil Gadzinski, Tianjin Huang, Mykola Pechenizkiy, Decebal Constantin Mocanu
Deep Learning Deep Neural Network Sparse Neural Network Dense Model

August 8, 2024

Unveiling the Power of Sparse Neural Networks for Feature Selection
Zahra Atashgahi, Tennison Liu, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu, Mihaela van der Schaar
Feature Selection Real Power Sparse Neural Network Dense Network Dynamic Sparse Training

May 24, 2024

Sparse maximal update parameterization: A holistic approach to sparse training dynamics
Nolan Dey, Shane Bergsma, Joel Hestness
Many Sparse Sparse Model Sparse Neural Network Dense Network Holistic Approach Dynamic Sparse Training

April 29, 2024

Learning a Sparse Neural Network using IHT
Saeed Damadi, Soroush Zolfaghari, Mahdi Rezaie, Jinglai Shen
Gradient Descent Sparse Neural Network Sparse Optimization Hard Thresholding Sparse Solution Technique

February 2, 2024

No Free Prune: Information-Theoretic Barriers to Pruning at Initialization
Tanishq Kumar, Kevin Luo, Mark Sellke
Mutual Information Edge Pruning Information Theoretic Sparse Network New Initialization Sparse Neural Network Sparse Subnetworks Dense Model

December 3, 2023

Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
Can Jin, Tianjin Huang, Yihua Zhang, Mykola Pechenizkiy, Sijia Liu, Shiwei Liu, Tianlong Chen
Model Pruning Sparse Neural Network Model Sparsification

October 28, 2023

Linear Mode Connectivity in Sparse Neural Networks
Luke McDermott, Daniel Cummings
Synthetic Data Sparse Network Sparse Neural Network Magnitude Pruning Synthetic Network Linear Mode Connectivity

September 30, 2023

The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks
Cameron Shinn, Collin McCarthy, Saurav Muralidharan, Muhammad Osama, John D. Owens
Sparsity Increase Sparse Neural Network Sparsity Pattern Hardware Constraint Sparse Kernel

August 22, 2023

Vision-Based Intelligent Robot Grasping Using Sparse Neural Network
Priya Shukla, Vandana Kushwaha, G C Nandi
Robotic Grasping Sparse Neural Network Grasping Dataset Intelligent Robot Grasping

July 14, 2023

Learning Sparse Neural Networks with Identity Layers
Mingjian Ni, Guangyao Chen, Xiawu Zheng, Peixi Peng, Li Yuan, Yonghong Tian
Sparse Network Sparse Neural Network Network Sparsity Sparsity Regularization Sparse Network Training

July 1, 2023

Sparse-Input Neural Network using Group Concave Regularization
Bin Luo, Susan Halabi
Feature Selection Sparse Neural Network Group Lasso Group Sparse

May 28, 2023

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers
Zahra Atashgahi, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu
Training Data Transformer Megatron Decepticons Sparse Neural Network Sparse Training Efficient Time Series Forecasting Forecasting Analysis Sparsity Tradeoff Adaptive Sparsity

May 26, 2023

Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations
Elia Cunegatti, Matteo Farina, Doina Bucur, Giovanni Iacca
Graph Representation Topology Problem Sparse Neural Network Pruning Framework Topological Metric Graph Encoding

May 24, 2023

Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference
Insung Kong, Dongyoon Yang, Jongjin Lee, Ilsang Ohn, Gyuseung Baek, Yongdai Kim
Bayesian Neural Network Posterior Inference Sparse Neural Network Theoretical Guarantee McMc Algorithm

May 22, 2023

Multiclass classification for multidimensional functional data through deep neural networks
Shuoyang Wang, Guanqun Cao
Deep Neural Network Multi Class Classification Sparse Neural Network Multiclass Classification High Dimensional Feature Functional Deep Neural Network Functional Data

May 18, 2023

Learning Activation Functions for Sparse Neural Networks
Mohammad Loni, Aditya Mohan, Mehdi Asadi, Marius Lindauer
Activation Function DNN Model Sparse Network Sparse Neural Network Sparse Activation

April 27, 2023

JaxPruner: A concise library for sparsity research
Joo Hyung Lee, Wonpyo Park, Nicole Mitchell, Jonathan Pilault, Johan Obando-Ceron, Han-Byul Kim, Namhoon Lee, Elias Frantar, Yun Long, Amir Yazdanbakhsh, Shivani Agrawal, Suvinay Subramanian, Xin Wang, Sheng-Chun Kao, Xingyao Zhang, Trevor Gale, Aart Bik, Woohyun Han, Milen Ferev, Zhonglin Han, Hong-Seok Kim, Yann Dauphin, Gintare Karolina Dziugaite, Pablo Samuel Castro, Utku Evci
Sparse Neural Network Sparse Training

April 21, 2023

Effective Neural Network $L_0$ Regularization With BinMask
Kai Jia, Martin Rinard
Regularization Model Sparse Neural Network Binary Mask Effective Neural

April 10, 2023

Bayesian optimization for sparse neural networks with trainable activation functions
Mohamed Fakhfakh, Lotfi Chaari
Bayesian Optimization Activation Function Sparse Neural Network