Gradient Accumulation

Gradient accumulation (GA) is a technique used in deep learning to train large models on limited GPU memory by accumulating gradients from multiple smaller batches before performing a single weight update. Current research focuses on applying GA to various architectures, including vision transformers (like Swin Transformers) and transformer-based models for natural language processing, as well as within specific optimization algorithms such as Adam. While GA can improve training efficiency for some models, its effectiveness varies depending on the architecture and task, with some studies showing performance degradation or increased training time. The overall goal is to enable training of larger and more complex models without being constrained by hardware limitations, impacting fields like medical image analysis and histopathology where large datasets are common.

Papers

January 3, 2024

Enhancing Generalization of Invisible Facial Privacy Cloak via Gradient Accumulation
Xuannan Liu, Yaoyao Zhong, Weihong Deng, Hongzhi Shi, Xingchen Cui, Yunfeng Yin, Dongchao Wen
Face Recognition Enhancing Generalization Black Box Face Recognition Gradient Accumulation

July 31, 2023

Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique
Sanad Aburass, Osama Dorgham
Vision Transformer Transformer Based Model Swin Transformer Performance Evaluation Gradient Accumulation

May 31, 2023

Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Training
Yijia Zhang, Yibo Han, Shijie Cao, Guohao Dai, Youshan Miao, Ting Cao, Fan Yang, Ningyi Xu
Memory Trace Targeted Activation Penalty Neural Network Training Gradient Episodic Memory Gradient Accumulation

March 8, 2023

Fast offset corrected in-memory training
Malte J. Rasch, Fabio Carta, Omebayode Fagbohungbe, Tayfun Gokmen
Memory Efficient Memory Computing Memory Efficient Training Memory Deep Learning Offset Building Model Gradient Accumulation

July 31, 2022

BYOLMed3D: Self-Supervised Representation Learning of Medical Videos using Gradient Accumulation Assisted 3D BYOL Framework
Siladittya Manna, Rakesh Dey, Souvik Chakraborty
Self Supervised Self Supervised Representation Learning Self Supervised Pre Training Method Medical Video Gradient Accumulation

March 8, 2022

End-to-end Multiple Instance Learning with Gradient Accumulation
Axel Andersson, Nadezhda Koriakina, Nataša Sladoje, Joakim Lindblad
Self Supervised Learning Pre Trained Model Multiple Instance Learning Multiple Instance Gradient Accumulation

Gradient Accumulation

Papers

Enhancing Generalization of Invisible Facial Privacy Cloak via Gradient Accumulation

Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique

Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Training

Fast offset corrected in-memory training

BYOLMed3D: Self-Supervised Representation Learning of Medical Videos using Gradient Accumulation Assisted 3D BYOL Framework

End-to-end Multiple Instance Learning with Gradient Accumulation