Sparse Local Model

Sparse local models aim to improve the efficiency and performance of machine learning models by reducing the number of parameters, focusing on only the most relevant ones for a given task or data subset. Current research emphasizes developing efficient training algorithms for various sparse architectures, including those employing structured sparsity (e.g., block sparsity, 1xN sparsity) and adaptive sparsity techniques that adjust the sparsity level during training. This focus on sparsity is driven by the need for reduced computational cost, memory footprint, and improved performance in resource-constrained environments, such as federated learning and privacy-preserving deep learning, as well as accelerating model adaptation and personalization.

Papers

October 1, 2024

SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models
Juan Pablo Muñoz, Jinjie Yuan, Nilesh Jain
Large Language Model Foundation Model Parameter Efficient Fine Tuning Large Pre Trained Model Efficient Model Adaptation Sparse Local Model

September 13, 2024

S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training
Yuezhou Hu, Jun Zhu, Jianfei Chen
Deep Neural Network High Efficiency Many Sparse Sparse Training Sparse Mask Sparse Local Model Meta Pruning

October 10, 2023

SUBP: Soft Uniform Block Pruning for 1xN Sparse CNNs Multithreading Acceleration
Jingyang Xiang, Siqi Li, Jun Chen, Shipeng Bai, Yukai Ma, Guang Dai, Yong Liu
Sparsity Increase Structured Sparsity High Sparsity Sparse Local Model Sparse CNN Block Wise Pruning

May 4, 2023

Efficient Personalized Federated Learning via Sparse Model-Adaptation
Daoyuan Chen, Liuyi Yao, Dawei Gao, Bolin Ding, Yaliang Li
Sparse Model Model Sparsity Sparse Local Model Sparse Update

January 30, 2023

Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models
Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis
Differential Privacy Private Stochastic Gradient Descent Equivariant Map Equivariant Convolutional Neural Network Sparse Local Model

August 4, 2022

ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity
Xinchi Qiu, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, Nicholas Donald Lane
Device Training Sparse Local Model Sparse Operation

June 1, 2022

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training
Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, Dacheng Tao
Heterogeneous Client Sparse Mask Sparse Local Model Decentralized Recommender System

April 16, 2022

Beyond L1: Faster and Better Sparse Models with skglm
Quentin Bertrand, Quentin Klopfenstein, Pierre-Antoine Bannier, Gauthier Gidel, Mathurin Massias
Many Sparse Linear Model Fast Algorithm Coordinate Descent Sparse Local Model Lagrange 1 Non Convex Penalty

April 9, 2022

Channel Pruning In Quantization-aware Training: An Adaptive Projection-gradient Descent-shrinkage-splitting Method
Zhijian Li, Jack Xin
Practical Method Quantization Aware Training Channel Pruning Shrinkage Estimator Gradient Projection Sparse Local Model Channel Sparsity Sparse Subspace

January 27, 2022

Achieving Personalized Federated Learning with Sparse Local Models
Tiansheng Huang, Shiwei Liu, Li Shen, Fengxiang He, Weiwei Lin, Dacheng Tao
Sparse Mask Sparse Local Model Sparse to Sparse Training

December 27, 2021

Sparsest Univariate Learning Models Under Lipschitz Constraint
Shayan Aziznejad, Thomas Debarre, Michael Unser
Continuous Time Regression Problem Lipschitz Network Sparse Local Model Regression Framework Lipschitz Constraint