Continuous Gradient

Continuous gradient methods are central to optimization problems across diverse fields, aiming to efficiently find minima of complex functions by iteratively following gradient directions. Current research focuses on improving gradient estimation and optimization algorithms, including addressing issues like noisy gradients in neural networks, escaping local optima in graph neural network attacks, and developing parameter-free or adaptive step-size strategies for gradient descent. These advancements enhance the efficiency and robustness of optimization, impacting areas such as machine learning, computer vision, and quantum computing by enabling faster training and improved model performance.

Papers

June 29, 2024

Axiomatization of Gradient Smoothing in Neural Networks
Linjiang Zhou, Xiaochuan Shi, Chao Ma, Zepeng Wang
Neural Network Axiomatic Approach Continuous Gradient Gradient Smoothing

June 19, 2024

AGSOA:Graph Neural Network Targeted Attack Based on Average Gradient and Structure Optimization
Yang Chen, Bin Zhou
Adversarial Attack Gradient Information Gradient Attack Continuous Gradient Structural Optimization

November 15, 2023

Non-Uniform Smoothness for Gradient Descent
Albert S. Berahas, Lindon Roberts, Fred Roosta
Gradient Descent Non Uniform Continuous Gradient

August 1, 2023

Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example
Ph. L. Toint
Natural Gradient Single Example Inverse Divergence Model Parameter Continuous Gradient Adam Algorithm

April 27, 2023

An Empirical Comparison of Optimizers for Quantum Machine Learning with SPSA-based Gradients
Marco Wiedmann, Marc Hölle, Maniraman Periyasamy, Nico Meyer, Christian Ufrecht, Daniel D. Scherer, Axel Plinge, Christopher Mutschler
Quantum Machine Learning Gradient Free Quantum Machine Better Optimizers Empirical Comparison Continuous Gradient Quantum K Mean Shallow Quantum Circuit

February 8, 2023

DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule
Maor Ivgi, Oliver Hinder, Yair Carmon
Gradient Descent Stochastic Gradient Descent Stochastic Gradient Convergence Guarantee Step Size Stochastic Convex Optimization Continuous Gradient

January 19, 2023

Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity -- the Strongly Convex Case
Tim Johnston, Iosif Lytras, Sotirios Sabanis
Optimization Problem Convex Set Langevin Monte Carlo Log Concave Continuous Gradient Lipschitz Gradient Euler Characteristic Hamiltonian Model

October 12, 2022

Sampling in Constrained Domains with Orthogonal-Space Variational Gradient Descent
Ruqi Zhang, Qiang Liu, Xin T. Tong
Gradient Flow Open Sampling Domain Constraint Continuous Gradient Orthogonal Gradient Manifold Sampling

September 26, 2022

Convergence rate of the (1+1)-evolution strategy on locally strongly convex functions with lipschitz continuous gradient and their monotonic transformations
Daiki Morinaga, Kazuto Fukuchi, Jun Sakuma, Youhei Akimoto
Convergence Rate Convex Function Evolution Strategy Derivative Free Optimization Continuous Gradient Numerical Black Box Optimization Canonical Lipschitz Mechanism

September 5, 2022

The Proxy Step-size Technique for Regularized Optimization on the Sphere Manifold
Fang Bai, Adrien Bartoli
Proximal Gradient Step Size Continuous Gradient Nuclear Norm Regularization

August 26, 2022

Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation
Jeffrey Willette, Seanie Lee, Bruno Andreis, Kenji Kawaguchi, Juho Lee, Sung Ju Hwang
Gradient Approximation Mini Batch Consistency Continuous Gradient

June 2, 2022

Accelerated first-order methods for convex optimization with locally Lipschitz continuous gradient
Zhaosong Lu, Sanyou Mei
Convex Optimization Non Convex First Order Method Continuous Gradient Constrained Convex Optimization Local Lipschitz

May 10, 2022

A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks
Mingrui Liu, Zhenxun Zhuang, Yunwei Lei, Chunyang Liao
Practical Algorithm Stochastic Gradient Descent Natural Gradient Communication Efficient Training Deep Communication Cost Slow Convergence Continuous Gradient

December 9, 2021

Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, Jifeng Dai
Contrastive Learning Self Supervised Learning Self Supervised Loss Function Feature Decorrelation Continuous Gradient

Continuous Gradient

Papers

Axiomatization of Gradient Smoothing in Neural Networks

AGSOA:Graph Neural Network Targeted Attack Based on Average Gradient and Structure Optimization

Non-Uniform Smoothness for Gradient Descent

Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example

An Empirical Comparison of Optimizers for Quantum Machine Learning with SPSA-based Gradients

DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule

Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity -- the Strongly Convex Case

Sampling in Constrained Domains with Orthogonal-Space Variational Gradient Descent

Convergence rate of the (1+1)-evolution strategy on locally strongly convex functions with lipschitz continuous gradient and their monotonic transformations

The Proxy Step-size Technique for Regularized Optimization on the Sphere Manifold

Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation

Accelerated first-order methods for convex optimization with locally Lipschitz continuous gradient

A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks

Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework