First Order Gradient

First-order gradient methods are fundamental optimization algorithms used to find the minimum of a function by iteratively moving in the direction of the negative gradient. Current research focuses on improving their efficiency and robustness, particularly for high-dimensional problems and non-convex functions like those encountered in training neural networks, exploring techniques such as variance reduction, adaptive sampling, and modifications to accelerate convergence. These advancements are crucial for tackling complex problems in machine learning, solving variational inequalities, and enhancing the performance of various applications, including image processing and scientific computing.

Papers

May 28, 2024

Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient
Hao Di, Haishan Ye, Yueling Zhang, Xiangyu Chang, Guang Dai, Ivor W. Tsang
Variance Reduction Composite Optimization Variance Reduction Technique First Order Gradient

February 24, 2024

Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimization
Jiaxin Deng, Junbiao Pang, Baochang Zhang, Tian Wang
Gradient Descent Sharpness Aware Minimization Node Feature Sharpness Sample Gradient Minimization Problem First Order Gradient

July 10, 2023

Invex Programs: First Order Algorithms and Their Convergence
Adarsh Barik, Suvrit Sra, Jean Honorio
Early Stage Convergence Gradient Method First Order Algorithm First Order Gradient

March 30, 2023

Fooling the Image Dehazing Models by First Order Gradient
Jie Gui, Xiaofeng Cong, Chengwei Peng, Yuan Yan Tang, James Tin-Yau Kwok
Image Dehazing Homogeneous Haze First Order Gradient ImageNet Attack

October 27, 2022

A Primal-dual Approach for Solving Variational Inequalities with General-form Constraints
Tatjana Chavdarova, Matteo Pagliardini, Tong Yang, Michael I. Jordan
Primal Dual Convergence Guarantee Variational Inequality Inequality Constraint First Order Gradient Hard Inequality Constraint

September 3, 2022

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One
John Chiang
Practical Method Hessian Matrix Gradient Method Order Gradient First Order Gradient

August 8, 2022

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks
Xin Liu, Wei Tao, Wei Li, Dazhi Zhan, Jun Wang, Zhisong Pan
Neural Network Nesterov Momentum Accelerated Gradient Method Anderson Acceleration Heavy Ball Accelerated Convergence Momentum Method First Order Gradient

May 20, 2022

SADAM: Stochastic Adam, A Stochastic Operator for First-Order Gradient-based Optimizer
Wei Zhang, Yu Bao
Stochastic Way First Order Adam Optimizer First Order Gradient Stochastic Operator

May 16, 2022

Optimizing the optimizer for data driven deep neural networks and physics informed neural networks
John Taylor, Wenyi Wang, Biswajit Bala, Tomasz Bednarz
Neural Network Deep Neural Network Raw Data Theoretical Physic Adam Optimizer Superior Optimizer Levenberg Marquardt First Order Gradient

First Order Gradient

Papers

Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient

Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimization

Invex Programs: First Order Algorithms and Their Convergence

Fooling the Image Dehazing Models by First Order Gradient

A Primal-dual Approach for Solving Variational Inequalities with General-form Constraints

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks

SADAM: Stochastic Adam, A Stochastic Operator for First-Order Gradient-based Optimizer

Optimizing the optimizer for data driven deep neural networks and physics informed neural networks