Full Batch Gradient Descent

Full-batch gradient descent (GD) is an optimization algorithm that updates model parameters using the entire dataset in each iteration, aiming for high accuracy and stable convergence. Current research focuses on mitigating its computational limitations, particularly for large datasets and complex models like Graph Neural Networks (GNNs), through techniques such as feature slicing, adaptive caching, and optimized distributed training schemes. These efforts aim to improve the scalability and efficiency of full-batch GD, making it a viable option for increasingly large-scale machine learning tasks while addressing challenges related to memory constraints and communication overhead. The resulting advancements have implications for various applications, including improved training of deep learning models and enhanced performance in specific domains like graph analysis.

Papers

August 21, 2024

Slicing Input Features to Accelerate Deep Learning: A Case Study with Graph Neural Networks
Zhengjia Xu, Dingyang Lyu, Jinghui Zhang
Deep Learning Graph Neural Network Case Study Large Scale Graph GNN Training Input Feature Full Batch Gradient Descent

August 1, 2024

CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reduction
Shuai Zhang, Zite Jiang, Haihang You
Design Principle Communication Cost Full Batch Gradient Descent

June 17, 2024

Just How Flexible are Neural Networks in Practice?
Ravid Shwartz-Ziv, Micah Goldblum, Arpit Bansal, C. Bayan Bruss, Yann LeCun, Andrew Gordon Wilson
Neural Network Convolutional Neural Network Practice Mode Deep Architecture Stochastic Learning Full Batch Gradient Descent

June 13, 2024

Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis
Xinrui Yang, Zhuohan Wang, Anthony Hu
Text to Image Model Text to Image Synthesis Prompt Optimization Text to Image Generation Model Prompt Generation Image Generation Task Textual Prompt Full Batch Gradient Descent Prompt Evolution

June 7, 2024

Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise
Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, Yaoqing Yang
Deep Neural Network Two Layer Neural Network Weight Matrix Large Learning Rate Gradient Noise Heavy Tail Full Batch Gradient Descent

June 6, 2024

Discrete error dynamics of mini-batch gradient descent for least squares regression
Jackie Lok, Rishi Sonthalia, Elizaveta Rebrova
Gradient Flow Least Square Mini Batch Gradient Descent Full Batch Gradient Descent Error Metric Discrete Dynamical System

April 7, 2024

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization
Roi Livni
Gradient Descent Sample Complexity Empirical Risk Minimization Generalization Error Stochastic Convex Optimization Full Batch Gradient Descent Smooth Stochastic Convex Optimization

January 22, 2024

The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization
Matan Schliserman, Uri Sherman, Tomer Koren
Strong Generalization Gradient Method Stochastic Convex Optimization Full Batch Gradient Descent Optimal Dependence Fat Shattering Dimension Pas Stochastic Gradient Descent

August 6, 2023

The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning
Nikhil Ghosh, Spencer Frei, Wooseok Ha, Bin Yu
Stochastic Gradient Descent Stochastic Gradient Sparsity Increase Feature Learning Autoencoder Model Neural Network Autoencoder Node Feature Sharpness Full Batch Gradient Descent

December 4, 2022

Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness
Ruggiero Seccia, Corrado Coppola, Giampaolo Liuzzi, Laura Palagi
Early Stage Convergence Natural Gradient Line Search Random Reshuffling Full Batch Gradient Descent Incremental Gradient

July 26, 2022

Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability
Zhouzi Li, Zixuan Wang, Jian Li
Extreme Edge Core Stability Linear Neural Network Modern Neural Network Full Batch Gradient Descent Pan Sharpening Sharpness Dynamic

June 15, 2022

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington
Stochastic Gradient Descent High Dimension Implicit Regularization Full Batch Gradient Descent Pas Stochastic Gradient Descent

April 26, 2022

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD
Konstantinos E. Nikolakakis, Farzin Haddadpour, Amin Karbasi, Dionysios S. Kalogerias
Convex Loss Non Convex Loss Function Risk Bound Full Batch Gradient Descent Norm Based Generalization Bound

November 11, 2021

Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs
Hesham Mostafa
Graph Neural Network Full Batch Gradient Descent Attention Based Graph Neural Network Numerical Aggregation

Full Batch Gradient Descent

Papers

Slicing Input Features to Accelerate Deep Learning: A Case Study with Graph Neural Networks

CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reduction

Just How Flexible are Neural Networks in Practice?

Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis

Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise

Discrete error dynamics of mini-batch gradient descent for least squares regression

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization

The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization

The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning

Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness

Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs