Gradient Normalization

Gradient normalization techniques aim to improve the efficiency and stability of training machine learning models by modifying the gradient updates used in optimization algorithms. Current research focuses on developing adaptive normalization methods, such as those integrated into Adam and AdamW optimizers, and analyzing their convergence properties under various smoothness assumptions for both convex and non-convex loss functions. These advancements are significant because they enhance training performance across diverse applications, including image generation, classification, and natural language processing, by addressing challenges like hyperparameter tuning and escaping suboptimal solutions.

Papers

October 29, 2024

Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate
Zhiqi Bu, Xiaomeng Jin, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Mingyi Hong
Large Language Model Constructive Approach Learning Rate Machine Unlearning Multi Task Optimization Gradient Normalization Gradient Difference

October 21, 2024

Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise
Tao Sun, Xinwang Liu, Kun Yuan
Performance Improvement Nonconvex Optimization Convergence Performance Heavy Tailed Noise Gradient Normalization

July 24, 2024

An Adaptive Gradient Regularization Method
Huixiu Jiang, Ling Yang, Yu Bao, Rutong Si
Adaptive Regularization Weight Normalization Gradient Normalization Gradient Vector

November 6, 2023

Parameter-Agnostic Optimization under Relaxed Smoothness
Florian Hübler, Junchi Yang, Xiang Li, Niao He
Gradient Descent Hyperparameter Tuning Uniform Convergence Parameter Free Gradient Normalization

August 17, 2023

Over-the-Air Computation Aided Federated Learning with the Aggregation of Normalized Gradient
Rongfei Fan, Xuming An, Shiyuan Zuo, Han Hu
Data Aggregation Air Federated Learning Air Computation Convex Loss Function Aggregated Gradient Private Convex Optimization Gradient Normalization

August 10, 2023

Normalized Gradients for All
Francesco Orabona
Black Box Natural Gradient Short Note Novel Concept Gradient Normalization

June 16, 2023

Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima
Dongkuk Si, Chulhee Yun
Sharpness Aware Minimization New Way Global Minimum Gradient Normalization Global Convergence Rate

May 25, 2023

SING: A Plug-and-Play DNN Learning Technique
Adrien Courtois, Damien Scieur, Jean-Michel Morel, Pablo Arias, Thomas Eboli
Strong Generalization Natural Gradient Plug and Play Better Optimizers Superior Optimizer Vocal Performance Gradient Normalization

November 4, 2021

GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
Vineeth S. Bhaskara, Tristan Aumentado-Armstrong, Allan Jepson, Alex Levinshtein
Generative Adversarial Network Feature Normalization GAN Loss Gradient Normalization