Sparse Neural Network

Sparse neural networks (SNNs) aim to improve the efficiency and interpretability of deep learning models by reducing the number of parameters while maintaining or even exceeding the performance of their dense counterparts. Current research focuses on developing novel pruning algorithms, exploring the interplay between data and model architecture in achieving sparsity, and investigating the impact of sparsity on training dynamics and generalization. This area is significant because SNNs offer the potential for reduced computational costs, improved energy efficiency, and enhanced model interpretability, leading to wider deployment of deep learning in resource-constrained environments and applications requiring explainability.

Papers

March 14, 2023

Vision-based route following by an embodied insect-inspired sparse neural network
Lu Yihe, Rana Alkhoury Maroun, Barbara Webb
Navigation Task Visual Input Sparse Model Sparse Neural Network Based Navigation

March 10, 2023

Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks
Zahra Atashgahi, Xuhao Zhang, Neil Kichler, Shiwei Liu, Lu Yin, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu
Feature Selection Sparse Neural Network

March 3, 2023

Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, Zhangyang Wang
Sparsity Increase Sparse Neural Network United State Efficient Sparse Sparse Solution Technique

February 6, 2023

Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers
Shiwei Liu, Zhangyang Wang
Critical Lesson Sparse Neural Network Sparse Training Dynamic Sparsity Sparse to Sparse Training Sparse Network Training Post Training Sparsity

January 28, 2023

Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
Tomer Galanti, Mengjia Xu, Liane Galanti, Tomaso Poggio
Deep Neural Network Generalization Bound Sparsity Increase Sparse Neural Network Sparse Deep Norm Based Generalization Bound

January 20, 2023

Accelerating and Compressing Deep Neural Networks for Massive MIMO CSI Feedback
Omar Erak, Hatem Abou-Zeid
Deep Neural Network Channel State Information Channel Estimation Sparse Neural Network Massive MIMO

January 2, 2023

Sparse neural networks with skip-connections for identification of aluminum electrolysis cell
Erlend Torje Berg Lundby, Haakon Robinsson, Adil Rasheed, Ivar Johan Halvorsen, Jan Tommy Gravdahl
Person Identification Sparse Neural Network Manufacturing System Nonlinear System Identification Sparse Input Modular Electrolysis Plant

December 25, 2022

Learning k-Level Sparse Neural Networks Using a New Generalized Weighted Group Sparse Envelope Regularization
Yehonathan Refael, Iftach Arbel, Wasim Huleihel
Structured Sparsity Sparse Neural Network

October 25, 2022

Gradient-based Weight Density Balancing for Robust Dynamic Sparse Training
Mathias Parger, Alexander Ertl, Paul Eibensteiner, Joerg H. Mueller, Martin Winter, Markus Steinberger
Balancing Weight Sparse Neural Network Sparse Subnetworks Dynamic Sparse Training High Sparsity Redistribution Mechanism

September 13, 2022

Sparse deep neural networks for modeling aluminum electrolysis dynamics
Erlend Torje Berg Lundby, Adil Rasheed, Ivar Johan Halvorsen, Jan Tommy Gravdahl
Deep Neural Network Many Sparse Regularization Model Sparse Model Sparse Neural Network Sparse Regularization Modular Electrolysis Plant

July 14, 2022

DASS: Differentiable Architecture Search for Sparse neural networks
Hamid Mousavi, Mohammad Loni, Mina Alibeigi, Masoud Daneshtalab
Sparse Network Sparse Neural Network Sparse Matrix Differentiable Architecture Search Sparse Convolution Sparse Operation

July 9, 2022

On the Robustness and Anomaly Detection of Sparse Neural Networks
Morgane Ayle, Bertrand Charpentier, John Rachwan, Daniel Zügner, Simon Geisler, Stephan Günnemann
Native Robustness Anomaly Detection Structured Sparsity Sparse Neural Network Sparse Subnetworks Robustness Problem

July 3, 2022

FasterAI: A Lightweight Library for Creating Sparse Neural Networks
Nathan Hubens
Pytorch Model AI Model Neural Network Compression Sparse Neural Network

July 2, 2022

A Structured Sparse Neural Network and Its Matrix Calculations Algorithm
Seyyed Mostafa Mousavi Janbeh Sarayi, Mansour Nikkhah Bahrami
Back Propagation Sparse Neural Network Sparse Matrix Large Matrix Matrix Computation

June 28, 2022

Hebbian Continual Representation Learning
Paweł Morawiecki, Andrii Krutsylo, Maciej Wołczyk, Marek Śmieja
Continual LEArning Class Incremental Learning Sparse Neural Network Hebbian Learning Continual Representation Learning

June 26, 2022

Training Your Sparse Neural Network Better with Any Mask
Ajay Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, Zhangyang Wang
Sparse Neural Network Mask Frozen DETR Sparse Training Sparse Mask

June 21, 2022

Renormalized Sparse Neural Network Pruning
Michael G. Rawson
Neural Network MNIST Dataset Sparse Neural Network Renormalization Group

June 18, 2022

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks
Mustafa Burak Gurbuz, Constantine Dovrolis
Continual LEArning Sparse Network Sparse Neural Network Trainable Parameter

June 17, 2022

The State of Sparse Training in Deep Reinforcement Learning
Laura Graesser, Utku Evci, Erich Elsen, Pablo Samuel Castro
Deep Reinforcement Learning Internal State Sparse Network Sparse Neural Network Sparse Training

June 2, 2022

Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
Mansheej Paul, Brett W. Larsen, Surya Ganguli, Jonathan Frankle, Gintare Karolina Dziugaite
Lottery Ticket Sparse Neural Network Dense Network Magnitude Pruning Dense Training Small Initialization Training Iteration Data Diet