Sparse Update

Sparse update methods in machine learning aim to improve training efficiency and resource utilization by updating only a subset of model parameters during training, rather than all parameters. Current research focuses on developing algorithms that strategically select which parameters to update, leveraging techniques like gradient magnitude analysis, structured sparsity, and randomized selection, often within the context of large language models (LLMs), vision transformers (ViTs), and federated learning. This approach offers significant benefits, including reduced memory footprint, faster training times, and lower communication costs, impacting various applications from on-device training to high-resolution image processing and large-scale model deployment.

Papers

July 3, 2024

SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning
Bac Nguyen, Stefan Uhlich, Fabien Cardinaux, Lukas Mauch, Marzieh Edraki, Aaron Courville
Fine Tuning Pre Trained Model Distribution Generalization Pre Trained Vision Language Model Sparse Update

June 25, 2024

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, Virginia Smith
Large Language Model Gradient Sparsification Projection Matrix Gradient Episodic Memory Grass Growth Sparse Update

October 7, 2023

Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks
Jules Berman, Benjamin Peherstorfer
Deep Network Sequential Learning Numerical Experiment Time Dependent Partial Differential Equation Deep Galerkin Method Sparse Update

October 5, 2023

Over-the-Air Federated Learning with Compressed Sensing: Is Sparsification Necessary?
Adrian Edin, Zheng Chen
Linear Compression Compressed Sensing Random Sparsification Air Federated Learning Signal Decomposition Sparse Update

July 19, 2023

TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge
Young D. Kwon, Rui Li, Stylianos I. Venieris, Jagmohan Chauhan, Nicholas D. Lane, Cecilia Mascolo
DNN Framework Device Training Energy Efficient Training Sparse Update Edge Data

May 4, 2023

Efficient Personalized Federated Learning via Sparse Model-Adaptation
Daoyuan Chen, Liuyi Yao, Dawei Gao, Bolin Ding, Yaliang Li
Sparse Model Model Sparsity Sparse Local Model Sparse Update

March 30, 2023

SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, Song Han
Activation Sparsity Deep Learning Workload Layerwise Sparsity Sparse Update High Resolution Vision Transformer

November 22, 2022

Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisation
Adrian Goldwaser, Hong Ge
Deep Network Feature Learning Deeper Network Reverse Graph Linearization Standard Gradient Descent Sparse Update

April 9, 2022

Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning
Daniel Becking, Heiner Kirchhoffer, Gerhard Tech, Paul Haase, Karsten Müller, Heiko Schwarz, Wojciech Samek
Convolutional Filter Gradient Update Adaptive Filter Sparse Update

November 18, 2021

Training Neural Networks with Fixed Sparse Masks
Yi-Lin Sung, Varun Nair, Colin Raffel
Neural Network Deep Neural Network Parameter Efficient Transfer Learning Sparse Mask Sparse Update

Sparse Update

Papers

SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients

Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks

Over-the-Air Federated Learning with Compressed Sensing: Is Sparsification Necessary?

TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge

Efficient Personalized Federated Learning via Sparse Model-Adaptation

SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer

Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisation

Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning

Training Neural Networks with Fixed Sparse Masks