Model Sparsity

Model sparsity focuses on reducing the number of parameters or activations in machine learning models to improve efficiency and resource utilization without significant performance loss. Current research explores various techniques, including weight pruning, activation sparsity, and the incorporation of sparsity-inducing regularizations during training, applied to diverse architectures like large language models and convolutional neural networks. This area is crucial for deploying large models on resource-constrained devices and improving the interpretability and robustness of machine learning systems, impacting both scientific understanding and practical applications across various domains.

Papers

August 26, 2024

Training-Free Activation Sparsity in Large Language Models
James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, Ben Athiwaratkun
Large Language Model Activation Sparsity Shallow ReLU Model Sparsity

March 17, 2024

IGANN Sparse: Bridging Sparsity and Interpretability with Non-linear Insight
Theodor Stoecker, Nico Hambauer, Patrick Zschech, Mathias Kraus
Inherent Interpretability Feature Selection Non Linear Generalized Additive Model Model Sparsity Sparsity Ratio

February 25, 2024

Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood
Rayen Dhahri, Alexander Immer, Betrand Charpentier, Stephan Günnemann, Vincent Fortuin
Neural Network Balancing Weight Marginal Likelihood Unstructured Sparsity Occam Algorithm Model Sparsification Model Sparsity Sparsity Prior

November 9, 2023

Compressed and Sparse Models for Non-Convex Decentralized Learning
Andrew Campbell, Hang Liu, Leah Woldemariam, Anna Scaglione
Sparse Model Decentralized Machine Learning Proximal Gradient Descent Model Sparsity Gradient Communication Decentralized Proximal

August 14, 2023

HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization
Patrick Glandorf, Timo Kaiser, Bodo Rosenhahn
Sparse Network Real World eXploitation Two Shift Regularization Loss Adaptive Regularization Efficient Sparse Model Sparsity

May 4, 2023

Efficient Personalized Federated Learning via Sparse Model-Adaptation
Daoyuan Chen, Liuyi Yao, Dawei Gao, Bolin Ding, Yaliang Li
Sparse Model Model Sparsity Sparse Local Model Sparse Update

April 11, 2023

Model Sparsity Can Simplify Machine Unlearning
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, Sijia Liu
Machine Unlearning Approximate Unlearning Model Sparsity

August 11, 2022

Quantized Adaptive Subgradient Algorithms and Their Applications
Ke Xu, Jianqiao Wangni, Yifan Zhang, Deheng Ye, Jiaxiang Wu, Peilin Zhao
Financial Application Learning Rate Mirror Descent Subgradient Method Model Sparsity Gradient Quantization

August 8, 2022

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints
Jose Gallego-Posada, Juan Ramirez, Akram Erraqabi, Yoshua Bengio, Simon Lacoste-Julien
Constrained Optimization Sparse Model Physical Constraint Appropriate Penalty Term Model Sparsity Sparsity Penalty Target Sparsity

July 6, 2022

Compositional Generalization in Grounded Language Learning via Induced Model Sparsity
Sam Spilsbury, Alexander Ilin
Compositional Generalization Internal Representation Language Navigation Grounded Language Model Sparsity

June 17, 2022

Sparse Double Descent: Where Network Pruning Aggravates Overfitting
Zheng He, Zeke Xie, Quanzhi Zhu, Zengchang Qin
Network Pruning Double Descent Model Sparsity Sparse Double Descent

February 10, 2022

Deadwooding: Robust Global Pruning for Deep Neural Networks
Sawinder Kaur, Ferdinando Fioretto, Asif Salekin
Deep Neural Network Native Robustness Adversarial Robustness Model Sparsity Global Pruning