Biased Gradient

Biased gradient estimation is a pervasive challenge in various machine learning contexts, particularly in distributed and meta-learning settings, where obtaining truly unbiased gradients is computationally expensive or even impossible. Current research focuses on analyzing the impact of biased gradients on algorithm convergence and generalization, developing methods to mitigate bias (e.g., through variance reduction techniques or novel gradient estimators), and establishing theoretical convergence guarantees for algorithms employing biased gradients. Understanding and addressing biased gradients is crucial for improving the efficiency and reliability of machine learning algorithms across diverse applications, from federated learning to reinforcement learning and multi-objective optimization.

Papers

August 25, 2024

Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learning
Wenxuan Zhou, Zhihao Qu, Shen-Huan Lyu, Miao Cai, Baoliu Ye
Model Training Split Learning Object Mask High Compression Sparse Mask Biased Gradient

August 20, 2024

Multi-level Monte-Carlo Gradient Methods for Stochastic Optimization with Biased Oracles
Yifan Hu, Jie Wang, Xin Chen, Niao He
Stochastic Optimization Monte Carlo Stochastic Optimization Problem Biased Gradient Conditional Stochastic Optimization

February 29, 2024

Parallel Momentum Methods Under Biased Gradient Estimations
Ali Beikmohammadi, Sarit Khirirat, Sindri Magnússon
Stochastic Gradient Gradient Estimation Unbiased Gradient Biased Gradient Momentum Method

February 5, 2024

Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation
Sobihan Surendran, Antoine Godichon-Baggioni, Adeline Fermanian, Sylvain Le Corff
Stochastic Gradient Descent Gradient Estimator Non Asymptotic Analysis Unbiased Gradient Biased Gradient

October 29, 2023

Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillation
Li Ding, Masrour Zoghi, Guy Tennenholtz, Maryam Karimzadehgan
Machine Learning Knowledge Distillation Neural Architecture Search Extension Study Meta Optimization Biased Gradient Adaptive EV Evaluator Adjuster Unit

October 28, 2023

Optimization of utility-based shortfall risk: A non-asymptotic viewpoint
Sumedh Gupte, Prashanth L. A., Sanjay P. Bhat
Optimization Purpose Non Asymptotic Biased Gradient

May 31, 2023

End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization
Shohei Taniguchi, Masahiro Suzuki, Yusuke Iwasawa, Yutaka Matsuo
Training Data Gibbs Sampler Debiased Contrastive Biased Gradient Deep Boltzmann

March 16, 2023

GLASU: A Communication-Efficient Algorithm for Federated Learning with Vertically Distributed Graph Data
Xinwei Zhang, Mingyi Hong, Jie Chen
Graph Structured Data Graph Data GNN Model Communication Efficient Model Splitting GNN Backbone Biased Gradient

December 24, 2022

When Do Curricula Work in Federated Learning?
Saeed Vahidian, Sreevatsank Kadaveru, Woonjoon Baek, Weijia Wang, Vyacheslav Kungurtsev, Chen Chen, Mubarak Shah, Bill Lin
Data Heterogeneity Sequential Learning Client Selection Local SGD Biased Gradient

December 23, 2022

Graph Federated Learning with Hidden Representation Sharing
Shuang Wu, Mingxuan Zhang, Yuantong Li, Carl Yang, Pan Li
Joint Representation Gradient Estimation Data Sharing Federated Graph Learning Biased Gradient

October 23, 2022

Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach
Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, Tianyi Chen
Multi Task Learning Multi Objective Optimization Multi Objective Convergent Algorithm Biased Gradient

June 13, 2022

Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients
Kyurae Kim, Jisu Oh, Jacob R. Gardner, Adji Bousso Dieng, Hongseok Kim
Stochastic Gradient Descent Variational Inference Unifying Framework Biased Gradient Gradient Variance

December 31, 2021

A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Xidong Feng, Bo Liu, Jie Ren, Luo Mai, Rui Zhu, Haifeng Zhang, Jun Wang, Yaodong Yang
Reinforcement Learning Meta Learning Theoretical Understanding Meta Reinforcement Learning Meta Gradient Biased Gradient Meta Rl Algorithm

December 14, 2021

Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learning
Yunhao Tang
Meta Reinforcement Learning Variance Reduction Gradient Estimation Score Function Unbiased Gradient Biased Gradient

Biased Gradient

Papers

Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learning

Multi-level Monte-Carlo Gradient Methods for Stochastic Optimization with Biased Oracles

Parallel Momentum Methods Under Biased Gradient Estimations

Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation

Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillation

Optimization of utility-based shortfall risk: A non-asymptotic viewpoint

End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization

GLASU: A Communication-Efficient Algorithm for Federated Learning with Vertically Distributed Graph Data

When Do Curricula Work in Federated Learning?

Graph Federated Learning with Hidden Representation Sharing

Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach

Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients

A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning

Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learning