Sparsity Tradeoff

Sparsity tradeoff research focuses on optimizing the balance between model size (sparsity) and performance in machine learning, aiming to reduce computational costs and memory footprint without sacrificing accuracy. Current efforts concentrate on developing efficient pruning algorithms for deep neural networks (DNNs), including transformers, leveraging techniques like second-order information, neuron saturation analysis, and combinatorial optimization to achieve superior sparsity-accuracy tradeoffs. This research is crucial for deploying large models on resource-constrained devices and improving the energy efficiency of machine learning applications across various domains, such as vision, language processing, and time series forecasting.

Papers

November 4, 2024

Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoders
Kola Ayonrinde
Supervised Autoencoder Sparse Autoencoders Sparse Feature Parameter Efficient Sparsity Adaptive Sparsity Sparsity Tradeoff

August 30, 2024

The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information
Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, Dan Alistarh
Second Order Sparse Recovery Sparse Deep Model Sparsification Sparsity Tradeoff

May 15, 2024

Unmasking Efficiency: Learning Salient Sparse Models in Non-IID Federated Learning
Riyasat Ohib, Bishal Thapaliya, Gintare Karolina Dziugaite, Jingyu Liu, Vince Calhoun, Sergey Plis
Many Sparse Sparse Model Sparse Subnetworks Sparse Federated Sparsity Tradeoff

March 12, 2024

Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
Simon Dufort-Labbé, Pierluca D'Oro, Evgenii Nikishin, Razvan Pascanu, Pierre-Luc Bacon, Aristide Baratin
Single Neuron Level Structured Pruning Saturation Effect Dead Neuron Network Sparsity Sparsity Tradeoff

August 31, 2023

The Quest of Finding the Antidote to Sparse Double Descent
Victor Quétu, Marta Milovanović
Deep Learning Model Sparsity Increase QueST Sparsity Tradeoff Sparse Double Descent

July 16, 2023

Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks
Haobo Song, Soumajit Majumder, Tao Lin
Vision Task Implicit Model Deep Equilibrium Model Sparsity Tradeoff Weight Tied Model

June 26, 2023

Gradient Descent Converges Linearly for Logistic Regression on Separable Data
Kyriakos Axiotis, Maxim Sviridenko
Gradient Descent Learning Rate Logistic Regression Separable Data Learning Guarantee Sparsity Tradeoff

May 28, 2023

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers
Zahra Atashgahi, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu
Training Data Transformer Megatron Decepticons Sparse Neural Network Sparse Training Efficient Time Series Forecasting Forecasting Analysis Sparsity Tradeoff Adaptive Sparsity

February 28, 2023

Fast as CHITA: Neural Network Pruning with Combinatorial Optimization
Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, Rahul Mazumder
Network Pruning Sparsity Constraint Pruning Framework Sparsity Tradeoff

October 7, 2022

Breaking BERT: Evaluating and Optimizing Sparsified Attention
Siddhartha Brahma, Polina Zablotskaia, David Mimno
Human Attention Many Sparse Sparsity Increase Transformer Layer Code BERT Sparsity Tradeoff

November 3, 2021

Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators
Marko Stamenovic, Nils L. Westhausen, Li-Chia Yang, Carl Jensen, Alex Pawlicki
Speech Enhancement Basic UniT Multi Modal Outer Arithmetic Block Neural Speech Enhancement Network Sparsity Sparsity Tradeoff Neural Accelerator

Sparsity Tradeoff

Papers

Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoders

The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information

Unmasking Efficiency: Learning Salient Sparse Models in Non-IID Federated Learning

Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons

The Quest of Finding the Antidote to Sparse Double Descent

Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks

Gradient Descent Converges Linearly for Logistic Regression on Separable Data

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers

Fast as CHITA: Neural Network Pruning with Combinatorial Optimization

Breaking BERT: Evaluating and Optimizing Sparsified Attention

Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators