Minibatch SGD

Minibatch stochastic gradient descent (SGD) is a widely used optimization algorithm that trains machine learning models by iteratively updating parameters based on small random subsets of the data. Current research focuses on understanding and mitigating the instability and error amplification that can arise from minibatch SGD's inherent noise, particularly in complex models and distributed settings like federated learning. This includes investigating the impact of minibatch size and exploring techniques like exponential moving averages to improve stability and generalization performance. These efforts aim to enhance the efficiency and reliability of training large-scale models across diverse applications, improving both the accuracy and scalability of machine learning.

Papers

October 17, 2023

Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression
Adam Block, Dylan J. Foster, Akshay Krishnamurthy, Max Simchowitz, Cyril Zhang
Stochastic Gradient Descent Autoregressive Model Behavior Cloning Autoregressive Text Generation Closed Loop Stability Butterfly Effect Error Simulation Minibatch SGD

October 2, 2023

Stability and Generalization for Minibatch SGD and Local SGD
Yunwen Lei, Tao Sun, Mingrui Liu
Strong Generalization Optimization Purpose Gradient Descent Core Stability Local SGD Parallel Optimization Minibatch SGD

August 23, 2023

When MiniBatch SGD Meets SplitFed Learning:Convergence Analysis and Performance Evaluation
Chao Huang, Geng Tian, Ming Tang
Convergence Analysis Gradient Norm Performance Evaluation Splitfed Learning Minibatch SGD

November 19, 2021

Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits
Hao Chen, Lili Zheng, Raed Al Kontar, Garvesh Raskutti
Gaussian Process Stochastic Gradient Descent Stochastic Gradient Convergence Guarantee Provable Benefit Mini Batch Gradient Descent Minibatch SGD

Minibatch SGD

Papers

Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression

Stability and Generalization for Minibatch SGD and Local SGD

When MiniBatch SGD Meets SplitFed Learning:Convergence Analysis and Performance Evaluation

Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits