Noisy Gradient

Noisy gradients, a pervasive challenge in optimization problems across machine learning, hinder the efficient training of models by introducing inaccuracies in the direction of parameter updates. Current research focuses on mitigating the effects of noisy gradients through various strategies, including improved gradient estimation techniques (e.g., using Wasserstein regression or noise-aware gradients), algorithmic modifications to enhance robustness (e.g., adapting accelerated gradient descent methods), and careful management of training parameters in federated learning settings. Addressing noisy gradients is crucial for improving the efficiency, accuracy, and robustness of machine learning models across diverse applications, from image synthesis and neural network pruning to federated learning and reinforcement learning.

Papers

October 25, 2024

Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training
Kristjan Greenewald, Yuancheng Yu, Hao Wang, Kai Xu
Generative Model Training Data Differential Privacy Adversarial Training Generative Modeling Privacy Policy Functional Mechanism Noisy Gradient

May 24, 2024

CAFe: Cost and Age aware Federated Learning
Sahan Liyanaarachchi, Kanchana Thilakarathna, Sennur Ulukus
Convergence Rate Hidden CoST Global Model Local Gradient Self Service CAFE Horizontal Federated Learning Convergence Theory Noisy Gradient

January 10, 2024

Score Distillation Sampling with Learned Manifold Corrective
Thiemo Alldieck, Nikos Kolotouros, Cristian Sminchisescu
Image Translation Image Diffusion Model Score Distillation Sampling Noisy Gradient

December 12, 2023

May the Noise be with you: Adversarial Training without Adversarial Examples
Ayoub Arous, Andres F Lopez-Lopera, Nael Abu-Ghazaleh, Ihsen Alouani
Adversarial Example Adversarial Training Loss Function Adversarial Learning Industrial Disturbing Noise Noisy Gradient

October 7, 2023

SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruning
Lei You, Hei Victor Cheng
Network Pruning Wasserstein Gradient Swap Distance Minimization Covariance Alignment Empirical Fisher Noisy Gradient

April 14, 2023

Exact Subspace Diffusion for Decentralized Multitask Learning
Shreya Wadehra, Roula Nassif, Stefan Vlaski
Decentralized Learning Multitask Learning Diffusion Network Consensus Mechanism Decentralized Gradient Noisy Gradient

February 15, 2023

Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumption
Jun Liu, Ye Yuan
Stochastic Gradient Descent Stochastic Gradient Gradient Norm Saddle Point Heavy Ball Noisy Gradient

February 10, 2023

Nesterov acceleration despite very noisy gradients
Kanan Gupta, Jonathan W. Siegel, Stephan Wojtowytsch
Stochastic Gradient Descent Low Rate ACCELERATION Gradient Estimation Accelerated Gradient Descent Noisy Gradient

June 14, 2022

How are policy gradient methods affected by the limits of control?
Ingvar Ziemann, Anastasios Tsiamis, Henrik Sandberg, Nikolai Matni
External Control Policy Gradient Continuum Limit Gradient Method Health Policy Recommendation Linear System Stochastic Policy Gradient Control Theoretic Noisy Gradient