Sparse Deep

Sparse deep learning focuses on creating deep neural networks with significantly fewer connections, reducing computational cost and memory requirements while maintaining accuracy. Current research emphasizes developing efficient algorithms for creating and training these sparse networks, including iterative pruning methods leveraging second-order information and $\ell_1$ regularization, and exploring their application in various architectures like convolutional neural networks and transformers. This field is crucial for deploying large-scale deep learning models on resource-constrained devices and improving the interpretability and efficiency of existing models, impacting both theoretical understanding and practical applications across diverse domains.

Papers

November 14, 2024

Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts
Mary Isabelle Wisell, Salimeh Yasaei Sekeh
Deep Neural Network Network Programming Distribution Shift Network Pruning Sparse Deep Benchmarking Deep Learning Fine Grained Personalized Guidance

August 30, 2024

The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information
Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, Dan Alistarh
Second Order Sparse Recovery Sparse Deep Model Sparsification Sparsity Tradeoff

August 10, 2024

Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks
Jianfei Li, Han Feng, Ding-Xuan Zhou
Convolutional Neural Network Many Sparse Convergence Analysis Sparse Feature Sparse Deep Sparse Coding

August 5, 2024

Sparse Deep Learning Models with the $\ell_1$ Regularization
Lixin Shen, Rui Wang, Yuesheng Xu, Mingsong Yan
Regularization Parameter Sparse Deep Sparsity Level Space Regularization

June 26, 2024

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression
Dongya Wu, Xin Li
Deep Neural Network Many Sparse Parameter Estimation Model Convergence Sparse Deep Efficient Sparse

May 27, 2024

Scorch: A Library for Sparse Deep Learning
Bobby Yan, Alexander J. Root, Trevor Gale, David Broman, Fredrik Kjolstad
Pytorch Model Easy to Use Library Sparse Deep Sparse Computation

March 29, 2024

Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World
Bowen Lei, Dongkuan Xu, Ruqi Zhang, Bani Mallick
Real World Cross Over Step Sparse Model Sample STEP Sparse Training Sparse Deep Uncertainty Aware Training Distribution Classifier

March 27, 2024

Causal-StoNet: Causal Inference for High-Dimensional Complex Data
Yaxin Fang, Faming Liang
Causal Inference High Dimension Sparse Deep

March 12, 2024

Abstracting Sparse DNN Acceleration via Structured Sparse Tensor Decomposition
Geonhwa Jeong, Po-An Tsai, Abhimanyu R. Bambhaniya, Stephen W. Keckler, Tushar Krishna
Sparse Deep Sparse Architecture Sparse DNN

March 1, 2024

Motif distribution and function of sparse deep neural networks
Olivia T. Zahn, Thomas L. Daniel, J. Nathan Kutz
Simple Function Connectivity Pattern Sparse Deep Network Motif

December 15, 2023

Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective
Yucong Dai, Gen Li, Feng Luo, Xiaolong Ma, Yongkai Wu
Edge Pruning Fairness Constraint Structured Pruning Model Fairness Bi Level Optimization Sparse Deep Fairness Optimization

November 15, 2023

Statistical learning by sparse deep neural networks
Felix Abramovich
Novel Regression Empirical Risk Minimization Statistical Learning Sparse Deep Discrete Logarithm Deep Neural Network Estimator

November 6, 2023

Testing RadiX-Nets: Advances in Viable Sparse Topologies
Kevin Kwak, Zack West, Hayden Jananthan, Jeremy Kepner
Sparse Network Network Topology Sparse Deep

October 23, 2023

Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network
Fuyuan Lyu, Xing Tang, Dugang Liu, Chen Ma, Weihong Luo, Liang Chen, Xiuqiang He, Xue Liu
High Dimensional Sparse Network Sparse Deep

October 5, 2023

Sparse Deep Learning for Time Series Data: Theory and Applications
Mingxuan Zhang, Yan Sun, Faming Liang
Financial Application Recurrent Neural Network Theoretical Understanding Many Sparse Autoregressive Model Time Series Data Prediction Uncertainty Sparse Deep

March 2, 2023

Sparse-penalized deep neural networks estimator under weak dependence
William Kengne, Modou Wade
Nonparametric Regression Sparse Deep Deep Neural Network Estimator

January 28, 2023

Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
Tomer Galanti, Mengjia Xu, Liane Galanti, Tomaso Poggio
Deep Neural Network Generalization Bound Sparsity Increase Sparse Neural Network Sparse Deep Norm Based Generalization Bound

December 11, 2022

Statistical guarantees for sparse deep learning
Johannes Lederer
Deep Learning Deep Network Many Sparse Sparsity Increase Statistical Guarantee Sparse Deep

July 27, 2022

Sparse Deep Neural Network for Nonlinear Partial Differential Equations
Yuesheng Xu, Taishan Zeng
Sparse Deep Sparse Regularization Multi Scale Sparse Sparsity Structure Adaptive Approximation

January 13, 2022

Examining and Mitigating the Impact of Crossbar Non-idealities for Accurate Implementation of Sparse Deep Neural Networks
Abhiroop Bhattacharjee, Lakshya Bhatnagar, Priyadarshini Panda
Global Impact Implementation Detail Crossbar Array Sparse Deep Structured Pruning Method