Gradient Method

Gradient methods are iterative optimization algorithms that aim to find the minimum (or maximum) of a function by repeatedly stepping in the direction of the negative (or positive) gradient. Current research focuses on improving the efficiency and robustness of these methods, particularly for non-convex problems arising in deep learning and other applications, exploring variations like stochastic gradient descent, adaptive methods (e.g., Adam), and incorporating second-order information or preconditioning techniques. These advancements are crucial for training complex models, enabling progress in fields like scientific machine learning and improving the performance of various machine learning tasks.

Papers

January 25, 2024

Transforming gradient-based techniques into interpretable methods
Caroline Mazini Rodrigues, Nicolas Boutry, Laurent Najman
Convolutional Neural Network Gradient Based Barzilai Borwein Technique Gradient Method Integrated Gradient Gradient Modulation

January 22, 2024

The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization
Matan Schliserman, Uri Sherman, Tomer Koren
Strong Generalization Gradient Method Stochastic Convex Optimization Full Batch Gradient Descent Optimal Dependence Fat Shattering Dimension Pas Stochastic Gradient Descent

January 16, 2024

GD doesn't make the cut: Three ways that non-differentiability affects neural network training
Siddharth Krishna Kumar
Neural Network Gradient Descent New Way Gradient Method Convex Function Non Convex Cut and Approximate Differentiable Framework

November 21, 2023

Acceleration and Implicit Regularization in Gaussian Phase Retrieval
Tyler Maunu, Martin Molina-Fructuoso
Gradient Descent Implicit Regularization Gradient Method Low Rate ACCELERATION Phase Retrieval Problem

October 23, 2023

Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens
Ross M. Clarke, José Miguel Hernández-Lobato
Second Order Optimization Process Gradient Method Approximate Curvature Second Order Algorithm

August 19, 2023

An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
Mohammad Sadegh Salehi, Subhadip Mukherjee, Lindon Roberts, Matthias J. Ehrhardt
Application Proficiency Bilevel Optimization Gradient Method Regularization Parameter Variational Regularization Bilevel Learning

August 18, 2023

Variance reduction techniques for stochastic proximal point algorithms
Cheik Traoré, Vassilis Apidopoulos, Saverio Salzo, Silvia Villa
Gradient Method Proximal Point Variance Reduction Technique Stochastic Proximal Point

July 26, 2023

Differentiable adaptive short-time Fourier transform with respect to the window length
Maxime Leiber, Yosra Marnissi, Axel Barrau, Mohammed El Badaoui
Gradient Method Short Time Fourier Transform Closed Form Differentiable Expression

July 18, 2023

PLiNIO: A User-Friendly Library of Gradient-based Methods for Complexity-aware DNN Optimization
Daniele Jahier Pagliari, Matteo Risso, Beatrice Alessandra Motetti, Alessio Burrello
Gradient Based Easy to Use Library Gradient Method Efficient Deep Efficient Neural Network

July 13, 2023

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima
Rishabh Dixit, Mert Gurbuzbalaban, Waheed U. Bajwa
Handwritten Trajectory Gradient Method Nonconvex Optimization Saddle Point Accelerated Gradient Method Local Minimum Nonconvex Function Smooth Nonconvex

July 11, 2023

Using Linear Regression for Iteratively Training Neural Networks
Harshad Khadilkar
Neural Network Back Propagation Linear Regression Gradient Method Iterative Neural Network Invertible Activation

July 10, 2023

Invex Programs: First Order Algorithms and Their Convergence
Adarsh Barik, Suvrit Sra, Jean Honorio
Early Stage Convergence Gradient Method First Order Algorithm First Order Gradient

June 26, 2023

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems
Kuangyu Ding, Jingyang Li, Kim-Chuan Toh
Deep Learning Application Proficiency Gradient Method Proximal Gradient Non Convex Objective Bregman Information Lipschitz Gradient Bregman Proximal

June 8, 2023

A Gradient-based Approach for Online Robust Deep Neural Network Training with Noisy Labels
Yifan Yang, Alec Koppel, Zheng Zhang
Noisy Label Gradient Based Gradient Method Robust Deep Scalable Training

May 30, 2023

Clip21: Error Feedback for Gradient Clipping
Sarit Khirirat, Eduard Gorbunov, Samuel Horváth, Rustem Islamov, Fakhri Karray, Peter Richtárik
Differential Privacy Gradient Descent Single CLIP Error Feedback Gradient Method Gradient Clipping

May 26, 2023

Local Convergence of Gradient Methods for Min-Max Games: Partial Curvature Generically Suffices
Guillaume Wang, Lénaïc Chizat
Nash Equilibrium Gradient Method Local Convergence Approximate Curvature Differentiable Game Min Max Game Local Nash Equilibrium

April 13, 2023

Improving Gradient Methods via Coordinate Transformations: Applications to Quantum Machine Learning
Pablo Bermejo, Borja Aizpurua, Roman Orus
Financial Application Gradient Descent Quantum Machine Learning Natural Gradient Gradient Method Optimization Algorithm Geometric Transformation Variational Representation

March 19, 2023

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization
Peiyuan Zhang, Jiaye Teng, Jingzhao Zhang
Gradient Descent Stochastic Gradient Descent Generalization Bound Objective Function Gradient Method Smooth Stochastic Convex Optimization

March 2, 2023

Tight Risk Bounds for Gradient Descent on Separable Data
Matan Schliserman, Tomer Koren
Gradient Descent Stochastic Gradient Descent Gradient Method Gradient Free Separable Data Risk Bound Smooth Loss Function

February 7, 2023

Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity
Risheng Liu, Yaohua Liu, Wei Yao, Shangzhi Zeng, Jin Zhang
Gradient Norm Gradient Method Bi Level Optimization Strong Convexity Class Wise Multiplier Single Loop Averaging Algorithm

Gradient Method

Papers

Transforming gradient-based techniques into interpretable methods

The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization

GD doesn't make the cut: Three ways that non-differentiability affects neural network training

Acceleration and Implicit Regularization in Gaussian Phase Retrieval

Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens

An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning

Variance reduction techniques for stochastic proximal point algorithms

Differentiable adaptive short-time Fourier transform with respect to the window length

PLiNIO: A User-Friendly Library of Gradient-based Methods for Complexity-aware DNN Optimization

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima

Using Linear Regression for Iteratively Training Neural Networks

Invex Programs: First Order Algorithms and Their Convergence

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

A Gradient-based Approach for Online Robust Deep Neural Network Training with Noisy Labels

Clip21: Error Feedback for Gradient Clipping

Local Convergence of Gradient Methods for Min-Max Games: Partial Curvature Generically Suffices

Improving Gradient Methods via Coordinate Transformations: Applications to Quantum Machine Learning

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Tight Risk Bounds for Gradient Descent on Separable Data

Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity