Gradient Method

Gradient methods are iterative optimization algorithms that aim to find the minimum (or maximum) of a function by repeatedly stepping in the direction of the negative (or positive) gradient. Current research focuses on improving the efficiency and robustness of these methods, particularly for non-convex problems arising in deep learning and other applications, exploring variations like stochastic gradient descent, adaptive methods (e.g., Adam), and incorporating second-order information or preconditioning techniques. These advancements are crucial for training complex models, enabling progress in fields like scientific machine learning and improving the performance of various machine learning tasks.

Papers

March 19, 2023

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization
Peiyuan Zhang, Jiaye Teng, Jingzhao Zhang
Gradient Descent Stochastic Gradient Descent Generalization Bound Objective Function Gradient Method Smooth Stochastic Convex Optimization

March 2, 2023

Tight Risk Bounds for Gradient Descent on Separable Data
Matan Schliserman, Tomer Koren
Gradient Descent Stochastic Gradient Descent Gradient Method Gradient Free Separable Data Risk Bound Smooth Loss Function

February 7, 2023

Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity
Risheng Liu, Yaohua Liu, Wei Yao, Shangzhi Zeng, Jin Zhang
Gradient Norm Gradient Method Bi Level Optimization Strong Convexity Class Wise Multiplier Single Loop Averaging Algorithm

January 30, 2023

Polynomial Preconditioning for Gradient Methods
Nikita Doikov, Anton Rodomanov
First Order Gradient Method Adaptive Preconditioner Non Convex Krylov Subspace Efficient Gradient

January 27, 2023

December 27, 2022

Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods
Alexander Shevchenko, Kevin Kögler, Hamed Hassani, Marco Mondelli
Supervised Autoencoder Gradient Method Lossy Compression Fundamental Limit Linear Autoencoder

November 10, 2022

Regression as Classification: Influence of Task Formulation on Neural Network Features
Lawrence Stewart, Francis Bach, Quentin Berthet, Jean-Philippe Vert
Neural Network Classification Code Novel Regression Cross Entropy Loss External Influence Gradient Method Gradient Based Optimization Two Layer ReLU Task Formulation

November 4, 2022

How Does Adaptive Optimization Impact Local Neural Network Geometry?
Kaiqi Jiang, Dhruv Malik, Yuanzhi Li
Gradient Method Adaptive Method Adaptive Algorithm Vanilla Gradient Descent Adaptive Optimization Method

October 26, 2022

Adaptive scaling of the learning rate by second order automatic differentiation
Frédéric de Gournay, Alban Gossard
Optimization Purpose Natural Gradient Learning Rate Second Order Gradient Method Automatic Differentiation Hessian Vector Product

October 13, 2022

Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks
Aditya Kumar Akash, Sixu Li, Nicolás García Trillos
Neural Network Unified Framework Gradient Method Model Fusion Wasserstein Barycenter Wasserstein Dictionary Linear Mode Connectivity

October 12, 2022

Rigorous dynamical mean field theory for stochastic gradient descent methods
Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, Lenka Zdeborova
Gradient Descent High Dimensional Stochastic Gradient Descent Gradient Flow Gradient Method Dynamical Mean Field Theory

September 19, 2022

Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks
Yunwen Lei, Rong Jin, Yiming Ying
Strong Generalization Gradient Descent Core Stability Gradient Method Shallow Neural Network

September 16, 2022

Stability and Generalization for Markov Chain Stochastic Gradient Methods
Puyu Wang, Yunwen Lei, Yiming Ying, Ding-Xuan Zhou
Strong Generalization Core Stability Empirical Risk Minimization Generalization Bound Markov Chain Gradient Method Optimal Convergence Rate

September 3, 2022

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One
John Chiang
Practical Method Hessian Matrix Gradient Method Order Gradient First Order Gradient

August 14, 2022

Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim, Chong You, Himanshu Jain, Andreas Veit, Rob Fergus, Sanjiv Kumar
Generative Model Training Data Knowledge Transfer Generalization Bound Gradient Method Efficient Framework

August 13, 2022

Riemannian accelerated gradient methods via extrapolation
Andi Han, Bamdev Mishra, Pratik Jawanpuria, Junbin Gao
Gradient Method Extrapolation Framework Riemannian Gradient Nesterov Momentum Riemannian Geometry Approach Acceleration Method

July 8, 2022

Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox
Abdurakhmon Sadiev, Dmitry Kovalev, Peter Richtárik
Primal Dual Gradient Method Local Gradient Concave Saddle Point Problem Communication Acceleration Mirror Prox

June 14, 2022

How are policy gradient methods affected by the limits of control?
Ingvar Ziemann, Anastasios Tsiamis, Henrik Sandberg, Nikolai Matni
External Control Policy Gradient Continuum Limit Gradient Method Health Policy Recommendation Linear System Stochastic Policy Gradient Control Theoretic Noisy Gradient

June 9, 2022

Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks
Huishuai Zhang, Da Yu, Yiping Lu, Di He
Neural Network Adversarial Attack Adversarial Example Neural Tangent Kernel Adversarial Noise Gradient Method Random Neural Network