Gradient Covariance

Gradient covariance, the covariance matrix of gradients during model training, is a key focus in improving the efficiency and accuracy of optimization algorithms, particularly in deep learning and complex systems modeling. Current research emphasizes leveraging gradient covariance information to enhance covariance estimation in heteroscedastic regression, develop memory-efficient adaptive regularization techniques, and understand the underlying structure of stochastic gradients for better optimization and generalization. These advancements are significant because they lead to improved model convergence, reduced computational costs, and a deeper understanding of the dynamics of training processes, impacting fields ranging from machine learning to fluid dynamics.

Papers

April 16, 2024

Analytical Approximation of the ELBO Gradient in the Context of the Clutter Problem
Roumen Nikolaev Popov
Context Information Mean Field Dense Clutter Gradient Approximation Evidence Lower Bound Numerical Approximation Variational Inference Problem Gradient Covariance

October 29, 2023

TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression
Megh Shukla, Mathieu Salzmann, Alexandre Alahi
New Framework Covariance Conditioning Covariance Estimation Negative Log Likelihood Heteroscedastic Regression Task Correlation Gradient Covariance

February 7, 2023

Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions
Vladimir Feinberg, Xinyi Chen, Y. Jennifer Sun, Rohan Anil, Elad Hazan
Covariance Matrix Interpretable Direction Adaptive Preconditioner Adaptive Regularization Untrusted Data Matrix Sketching Gradient Covariance

December 5, 2022

On the Overlooked Structure of Stochastic Gradients
Zeke Xie, Qian-Yuan Tang, Mingming Sun, Ping Li
Stochastic Gradient Gradient Noise Hidden Structure Gradient Covariance

July 28, 2022

Model Reduction for Nonlinear Systems by Balanced Truncation of State and Gradient Covariance
Samuel E. Otto, Alberto Padovan, Clarence W. Rowley
High Dimensional Nonlinear Dynamic Internal State Reduced Order Model Reduction Data Driven Reduced Order Galerkin Method Balanced Truncation Gradient Covariance

May 19, 2022

FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data
Mike He Zhu, Léna Néhale Ezzine, Dianbo Liu, Yoshua Bengio
Raw Data Domain Shift Problem Gradient Covariance

February 21, 2022

MSTGD:A Memory Stochastic sTratified Gradient Descent Method with an Exponential Convergence Rate
Aixiang, Chen, Jinting Zhang, Zanbo Zhang, Zhihong Li
Natural Gradient Variance Reduction Exponential Convergence Rate Sample Gradient Stratified Sampling Gradient Covariance

Gradient Covariance

Papers

Analytical Approximation of the ELBO Gradient in the Context of the Clutter Problem

TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression

Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions

On the Overlooked Structure of Stochastic Gradients

Model Reduction for Nonlinear Systems by Balanced Truncation of State and Gradient Covariance

FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data

MSTGD:A Memory Stochastic sTratified Gradient Descent Method with an Exponential Convergence Rate