Standard Gradient Descent

Standard gradient descent (SGD) is a fundamental optimization algorithm used to train machine learning models, aiming to minimize a loss function by iteratively adjusting model parameters. Current research focuses on improving SGD's efficiency and robustness, exploring variations like adaptive batch size and step size methods, and incorporating regularization techniques to enhance generalization and mitigate issues like overfitting and sensitivity to noisy data. These advancements are crucial for training increasingly complex models on large datasets, impacting diverse applications from image recognition to solving complex scientific problems.

Papers

October 17, 2024

On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery
Renpu Liu, Ruida Zhou, Cong Shen, Jing Yang
Transformer Megatron Decepticons Context Learning Gradient Descent Different Context Learned Optimizers Sparse Recovery Standard Gradient Descent

September 25, 2024

Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization
Francisco Aguilera-Martínez, Fernando Berzal
Training Data Differential Privacy Stochastic Gradient Descent Regularization Loss Standard Gradient Descent Novel Regularization

July 22, 2024

Regression under demographic parity constraints via unlabeled post-processing
Evgenii Chzhen, Mohamed Hebiri, Gayane Taturyan
Novel Regression Post Processing Sensitive Attribute Standard Gradient Descent Demographic Parity

February 7, 2024

AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
Petr Ostroukhov, Aigerim Zhumabayeva, Chulu Xiang, Alexander Gasnikov, Martin Takáč, Dmitry Kamzolov
Stochastic Gradient Descent Step Size Standard Gradient Descent Adaptive Batch Size

March 31, 2023

Per-Example Gradient Regularization Improves Learning Signals from Noisy Data
Xuran Meng, Yuan Cao, Difan Zou
Gradient Descent Noisy Data Regularization Technique Gradient Regularization Standard Gradient Descent Label Regularization Learning Signal

January 31, 2023

Patch Gradient Descent: Training Neural Networks on Very Large Images
Deepak K. Gupta, Gowreesh Mago, Arnav Chavan, Dilip K. Prasad
Neural Network Large Scale Patch Based Gradient Update Traditional CNNs Standard Gradient Descent

November 22, 2022

Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisation
Adrian Goldwaser, Hong Ge
Deep Network Feature Learning Deeper Network Reverse Graph Linearization Standard Gradient Descent Sparse Update

October 7, 2022

Scaling Forward Gradient With Local Losses
Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton
Loss Function Backpropagation Free Standard Gradient Descent Forward Gradient Gradient Based Solver Local Loss

October 6, 2022

Double Averaging and Gradient Projection: Convergence Guarantees for Decentralized Constrained Optimization
Firooz Shahriari-Mehr, Ashkan Panahi
Optimal Transport Optimization Problem Convergence Guarantee Global Optimality Gradient Projection Standard Gradient Descent Self Averaging

July 24, 2022

Can we achieve robustness from data alone?
Nikolaos Tsilivis, Jingtong Su, Julia Kempe
Adversarial Attack Native Robustness Raw Data Meta Learning Robust Classification Standard Gradient Descent

July 16, 2022

Adaptive Sketches for Robust Regression with Importance Sampling
Sepideh Mahabadi, David P. Woodruff, Samson Zhou
Stochastic Gradient Descent Importance Sampling Robust Regression Standard Gradient Descent Sketch and Lift

July 13, 2022

Implicit regularization of dropout
Zhongwang Zhang, Zhi-Qin John Xu
Strong Generalization Structured Dropout Implicit Regularization Regularization Method Standard Gradient Descent

June 13, 2022

Towards Understanding Sharpness-Aware Minimization
Maksym Andriushchenko, Nicolas Flammarion
Sharpness Aware Minimization Generalization Property Improved Generalization Standard Gradient Descent

March 3, 2022

Improvements to Gradient Descent Methods for Quantum Tensor Network Machine Learning
Fergus Barratt, James Dborin, Lewis Wright
Gradient Descent Large Relevance Improvement Tensor Network Regularization Technique Standard Gradient Descent

February 26, 2022

Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations
Metehan Cekic, Can Bakiskan, Upamanyu Madhow
Deep Neural Network Adversarial Training Adversarial Perturbation Many Sparse Brain Inspired Standard Gradient Descent

February 13, 2022

Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems
Levon Nurbekyan, Wanzhou Lei, Yunan Yang
Natural Gradient Optimization Problem Nonconvex Optimization Natural Gradient Descent Standard Gradient Descent Numerical Linear Algebra

January 25, 2022

Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition
Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Valérie Garès, Mounir Haddou, Quang Huy Tran
Neural Network Fisher Information Natural Gradient Descent Learning Benchmark Kronecker Product Efficient Approximation Standard Gradient Descent Empirical Fisher

December 4, 2021

Optimization-Based Separations for Neural Networks
Itay Safran, Jason D. Lee
Neural Network Deep Neural Network Gradient Descent Separation Performance Standard Gradient Descent Depth Separation

November 25, 2021

Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)
Umangi Jain, Harish G. Ramaswamy
Deep Learning Deep Neural Network Gradient Descent Hidden Layer Financial Success Standard Gradient Descent