Gradient Sparsification

Gradient sparsification aims to accelerate and scale machine learning training by reducing the communication volume of gradient updates, particularly crucial in distributed and federated learning settings. Current research focuses on developing efficient sparsification algorithms, often incorporating Bayesian inference, compressed sensing techniques, and dynamic thresholding to select the most informative gradient components, with applications spanning various model architectures including ResNets, Transformers, and LLMs. These advancements offer significant potential for improving the efficiency and scalability of training large models, impacting both the speed of scientific discovery and the deployment of AI in resource-constrained environments.

Papers

October 15, 2024

Age-of-Gradient Updates for Federated Learning over Random Access Channels
Yu Heng Wu, Houman Asgari, Stefano Rini, Andrea Munari
Gradient Compression Gradient Update Channel Quality Random Access Gradient Sparsification Slot Based

September 23, 2024

Novel Gradient Sparsification Algorithm via Bayesian Inference
Ali Bereyhi, Ben Liang, Gary Boudreau, Ali Afana
Bayesian Inference Random Sparsification Sparse Mask Gradient Sparsification Top K Sparsification

July 25, 2024

Sparse Incremental Aggregation in Multi-Hop Federated Learning
Sourav Mukherjee, Nasrin Razmi, Armin Dekorsy, Petar Popovski, Bho Matthiesen
Gradient Sparsification Sparsification Method Network Aggregation Incremental Aggregation

July 8, 2024

CONGO: Compressive Online Gradient Optimization
Jeremy Carleton, Prathik Vijaykumar, Divyanshu Saxena, Dheeraj Narasimha, Srinivas Shakkottai, Aditya Akella
Application Proficiency Compressed Sensing Online Convex Optimization Microservice Architecture Gradient Sparsification Sparse Gradient Compressive Learning

June 25, 2024

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, Virginia Smith
Large Language Model Gradient Sparsification Projection Matrix Gradient Episodic Memory Grass Growth Sparse Update

May 27, 2024

Gradient Compressed Sensing: A Query-Efficient Gradient Estimator for High-Dimensional Zeroth-Order Optimization
Ruizhong Qiu, Hanghang Tong
High Dimension Zeroth Order Gradient Sparsification Sparse Gradient

March 6, 2024

SPEAR:Exact Gradient Inversion of Batches in Federated Learning
Dimitar I. Dimitrov, Maximilian Baader, Mark Niklas Müller, Martin Vechev
Gradient Inversion Attack Large Batch Low Rank Structure Gradient Inversion Gradient Sparsification Computational Reconstruction

February 21, 2024

Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning
Daegun Yoon, Sangyoon Oh
Deep Learning Potential Scalability Gradient Sparsification

December 29, 2023

RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation
Weiying Xie, Zixuan Wang, Jitao Ma, Daixun Li, Yunsong Li
Remote Sensing Remote Sensing Image Gradient Compression Gradient Sparsification Gradient Communication

November 14, 2023

Sparsity-Preserving Differentially Private Training of Large Embedding Models
Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
Full Model Differential Privacy Stochastic Gradient Descent User Privacy Private Model Training Gradient Sparsification

October 2, 2023

MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training
Daegun Yoon, Sangyoon Oh
Multiplicative Size Scaling Random Sparsification Deep Neural Network Training Gradient Sparsification Aggregated Gradient Micro Level

September 6, 2023

A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness
Ze Peng, Lei Qi, Yinghuan Shi, Yang Gao
Adversarial Robustness Theoretical Understanding Sparsity Increase Activation Sparsity Flat Minimum Gradient Sparsification Sparsity Ratio

July 19, 2023

Gradient Sparsification For Masked Fine-Tuning of Transformers
James O' Neill, Sourav Dutta
Language Model Fine Tuning Transformer Megatron Decepticons Gradient Sparsification Modality Freezing Gradual Unfreezing

July 7, 2023

DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification
Daegun Yoon, Sangyoon Oh
Multi Layer Gradient Norm Gradient Sparsification

May 15, 2023

Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks
Minyoung Huh, Brian Cheung, Pulkit Agrawal, Phillip Isola
Vector Quantization Gradient Sparsification Straight Through Estimator

April 9, 2023

Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy
Kang Wei, Jun Li, Chuan Ma, Ming Ding, Feng Shu, Haitao Zhao, Wen Chen, Hongbo Zhu
Differential Privacy Federated Learning Random Sparsification Gradient Sparsification

March 15, 2023

Efficient and Secure Federated Learning for Financial Applications
Tao Liu, Zhi Wang, Hui He, Wei Shi, Liangliang Lin, Wei Shi, Ran An, Chenhao Li
High Efficiency Financial Application Federated Learning Random Sparsification Gradient Sparsification Rate Reduction Sparsification Method

September 18, 2022

Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment
Daegun Yoon, Sangyoon Oh
Empirical Analysis Distributed Learning Gradient Sparsification Top K Sparsification Differentiable Sorting

February 5, 2022

Distributed Learning With Sparsified Gradient Differences
Yicheng Chen, Rick S. Blum, Martin Takac, Brian M. Sadler
LeArning Abstract Gradient Descent Nonconvex Optimization Gradient Sparsification Linear Convergence Rate

January 30, 2022

OptG: Optimizing Gradient-driven Criteria in Network Sparsity
Yuxin Zhang, Mingbao Lin, Mengzhao Chen, Fei Chao, Rongrong Ji
Sparse Subnetworks Gradient Sparsification High Sparsity Network Sparsity

Gradient Sparsification

Papers

Age-of-Gradient Updates for Federated Learning over Random Access Channels

Novel Gradient Sparsification Algorithm via Bayesian Inference

Sparse Incremental Aggregation in Multi-Hop Federated Learning

CONGO: Compressive Online Gradient Optimization

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients

Gradient Compressed Sensing: A Query-Efficient Gradient Estimator for High-Dimensional Zeroth-Order Optimization

SPEAR:Exact Gradient Inversion of Batches in Federated Learning

Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning

RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation

Sparsity-Preserving Differentially Private Training of Large Embedding Models

MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training

A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness

Gradient Sparsification For Masked Fine-Tuning of Transformers

DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification

Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks

Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy

Efficient and Secure Federated Learning for Financial Applications

Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment

Distributed Learning With Sparsified Gradient Differences

OptG: Optimizing Gradient-driven Criteria in Network Sparsity