Gradient Descent

Gradient descent is an iterative optimization algorithm used to find the minimum of a function by repeatedly taking steps proportional to the negative of the gradient. Current research focuses on improving its efficiency and robustness, particularly in high-dimensional spaces and with non-convex functions, exploring variations like stochastic gradient descent, proximal methods, and natural gradient descent, often within the context of deep learning models and other complex architectures. These advancements are crucial for training increasingly complex machine learning models and improving their performance in various applications, from image recognition to scientific simulations. A key area of investigation involves understanding and mitigating issues like vanishing/exploding gradients, overfitting, and the impact of data characteristics on convergence.

Papers

October 16, 2024

Efficient Optimization Algorithms for Linear Adversarial Training
Antônio H. RIbeiro, Thomas B. Schön, Dave Zahariah, Francis Bach
Adversarial Training Gradient Descent Convex Optimization Optimization Algorithm Ridge Regression

October 15, 2024

October 14, 2024

Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks
Binghui Li, Zhixuan Pan, Kaifeng Lyu, Jian Li
Neural Network Native Robustness Gradient Descent Stochastic Gradient Descent Implicit Bias Non Robust

October 12, 2024

Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networks
Zhenghao Xu, Yuqing Wang, Tuo Zhao, Rachel Ward, Molei Tao
Gradient Descent Matrix Factorization Nonconvex Optimization Linear Neural Network Anderson Acceleration Accelerated Gradient Small Initialization Linear Convergence Rate Planar N

October 11, 2024

Efficient line search for optimizing Area Under the ROC Curve in gradient descent
Jadon Fowler, Toby Dylan Hocking
Gradient Descent Binary Classification Change Point Detection ROC Curve Area Usage Line Search

October 10, 2024

October 8, 2024

October 6, 2024

Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets
Tianjian Li, Haoran Xu, Weiting Tan, Kenton Murray, Daniel Khashabi
Language Model Data Set Gradient Descent Stochastic Gradient Descent Low Resource Language Balancing Weight High Quality Data Upsampling Operation

October 3, 2024

October 2, 2024

Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context
Spencer Frei, Gal Vardi
Transformer Based Gradient Descent Benign Overfitting Linear Transformer General Transformer

October 1, 2024

Differentiable Interacting Multiple Model Particle Filtering
John-Joseph Brady, Yuhui Luo, Wenwu Wang, Víctor Elvira, Yunpeng Li
Gradient Descent Sequential Monte Carlo Differentiable Particle Filter

September 30, 2024

(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Number
Rossen Nenov, Daniel Haider, Peter Balazs
Neural Network Gradient Descent Natural Gradient Denoising Process Gradient Regularization Numerical Stability

Gradient Descent

Papers

Efficient Optimization Algorithms for Linear Adversarial Training

State-space models can learn in-context by gradient descent

Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent

Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

Phase retrieval: Global convergence of gradient descent with optimal sample complexity

Stability and Sharper Risk Bounds with Convergence Rate $O(1/n^2)$

Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networks

Efficient line search for optimizing Area Under the ROC Curve in gradient descent

Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?

Optimal Transportation by Orthogonal Coupling Dynamics

On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks

Memory-augmented Transformers can implement Linear First-Order Optimization Methods

Benign Overfitting for Regression with Trained Two-Layer ReLU Networks

Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets

From Optimization to Sampling via Lyapunov Potentials

Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression

Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context

Differentiable Interacting Multiple Model Particle Filtering

(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Number