Gradient Similarity

Gradient similarity, a measure of the resemblance between gradients of a loss function across different data points or models, is emerging as a crucial concept in machine learning optimization and data analysis. Current research focuses on leveraging gradient similarity to improve the efficiency and robustness of distributed and federated learning, enhance data valuation by identifying low-quality samples, and accelerate convergence through techniques like synthetic data shuffling and momentum-based error feedback. This work has significant implications for improving the scalability, accuracy, and interpretability of machine learning models across diverse applications, particularly in scenarios with limited data or high communication costs.

Papers

October 29, 2024

Gradient Distance Function
Hieu Le, Federico Stella, Benoit Guillard, Pascal Fua
Distance Function Shape Aware Gradient Similarity Unsigned Distance Function

September 22, 2024

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning
Dmitry Bylinkin, Kirill Degtyarev, Aleksandr Beznosikov
Early Stage Convergence High Similarity Hessian Matrix Clean Distribution Data Similarity Gradient Similarity

May 13, 2024

Data Valuation with Gradient Similarity
Nathaniel J. Evans, Gordon B. Mills, Guanming Wu, Xubo Song, Shannon McWeeney
Data Valuation Low Quality Gradient Similarity

August 16, 2023

Warped geometric information on the optimisation of Euclidean functions
Marcelo Hartmann, Bernardo Williams, Hanlin Yu, Mark Girolami, Alessandro Barp, Arto Klami
Optimization Process Generalized Geodesic Geometric Information Euclidean Space Gradient Similarity Metric Tensor

June 23, 2023

Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
Bo Li, Yasin Esfandiari, Mikkel N. Schmidt, Tommy S. Alstrøm, Sebastian U. Stich
LeArning Abstract Synthetic Data Early Stage Convergence Data Heterogeneity Shuffle Model Gradient Similarity

May 24, 2023

Momentum Provably Improves Error Feedback!
Ilyas Fatkhullin, Alexander Tyurin, Peter Richtárik
Error Feedback Exponential Convergence Rate Stochastic Nonconvex Momentum Transformer Gradient Similarity

September 29, 2022

Learning Gradient-based Mixup towards Flatter Minima for Domain Generalization
Danni Peng, Sinno Jialin Pan
LeArning Abstract Domain Generalization Unseen Domain Better Generalization Spatio Temporal Mixup Mechanism Flat Minimum Gradient Similarity

September 10, 2022

Adaptive Meta-learner via Gradient Similarity for Few-shot Text Classification
Tianyi Lei, Honghui Hu, Qiaoyang Luo, Dezhong Peng, Xu Wang
Meta Learning Shot Text Classification Gradient Similarity Optimization Based Meta Learning

June 9, 2022

On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data
Jianyu Wang, Rudrajit Das, Gauri Joshi, Satyen Kale, Zheng Xu, Tong Zhang
Heterogeneous Data Federated Averaging FedAvg Algorithm Gradient Similarity

May 25, 2022

On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Vincent Szolnoky, Viktor Andersson, Balazs Kulcsar, Rebecka Jörnsten
Neural Network Inherent Interpretability Regularization Model Explicit Regularization Regularization Property Gradient Similarity