Efficient Gradient

Efficient gradient methods aim to accelerate optimization processes by improving the computation and application of gradients, crucial for training complex models in machine learning. Current research focuses on developing novel algorithms like adjoint methods for diffusion models and alternating gradient descent approaches for minimax problems, as well as improving existing methods through techniques such as preconditioning and gradient clipping tailored to specific problem structures (e.g., those with multiple scales or relaxed smoothness). These advancements are significant because they enable faster training of large-scale models and improve the efficiency of various machine learning applications, including those involving deep learning, reinforcement learning, and optimal transport.

Papers

May 23, 2024

AdjointDEIS: Efficient Gradients for Diffusion Models
Zander W. Blasingame, Chen Liu
Diffusion Model Diffusion SDE Efficient Gradient

February 16, 2024

Fundamental Benefit of Alternating Updates in Minimax Optimization
Jaewook Lee, Hanseul Cho, Chulhee Yun
Complementary Benefit Minimax Optimization Gradient Descent Ascent Information System Update Stochastic Gradient Descent Ascent Lipschitz Gradient Efficient Gradient Extra Gradient Method

December 21, 2023

On Partial Optimal Transport: Revising the Infeasibility of Sinkhorn and Efficient Gradient Methods
Anh Duc Nguyen, Tuan Dung Nguyen, Quang Minh Nguyen, Hoang H. Nguyen, Lam M. Nguyen, Kim-Chuan Toh
Primal Dual Sinkhorn Algorithm Efficient Approximation Efficient Gradient

May 25, 2023

Differentiable Clustering with Perturbed Spanning Forests
Lawrence Stewart, Francis S Bach, Felipe Llinares López, Quentin Berthet
Semi Supervised Clustering Random Perturbation Efficient Gradient Differentiable Clustering

April 21, 2023

A Cubic-regularized Policy Newton Algorithm for Reinforcement Learning
Mizhaan Prajit Maniyar, Akash Mondal, Prashanth L. A., Shalabh Bhatnagar
Reinforcement Learning Policy Gradient Efficient Gradient Cubic Regularization

February 14, 2023

EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data
Michael Crawshaw, Yajie Bao, Mingrui Liu
Stochastic Gradient Heterogeneous Data Gradient Norm Uncertain Episode Length Gradient Clipping Gradient Episodic Memory Efficient Gradient

January 30, 2023

Polynomial Preconditioning for Gradient Methods
Nikita Doikov, Anton Rodomanov
First Order Gradient Method Adaptive Preconditioner Non Convex Krylov Subspace Efficient Gradient

March 1, 2022

When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee
Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, Tianbao Yang
Deep Learning Robust Optimization Distributionally Robust Optimization Value at Risk Partial AUC AUC Metric Efficient Gradient

November 4, 2021

Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scales
Jonathan Kelner, Annie Marsden, Vatsal Sharan, Aaron Sidford, Gregory Valiant, Honglin Yuan
Convex Optimization Accelerated Gradient Method Multiscale Problem Efficient Gradient