Stale Gradient

Stale gradient problems arise in distributed machine learning when model updates are based on outdated information, hindering efficient and accurate training. Current research focuses on mitigating this issue through techniques like corrector networks that adjust stale embeddings, asynchronous mini-batching strategies that handle variable delays, and adaptive algorithms that selectively utilize fresh and stale updates in federated learning and other distributed settings. Addressing stale gradients is crucial for improving the scalability and efficiency of large-scale machine learning, impacting diverse applications from recommendation systems to federated learning across heterogeneous devices.

Papers

September 3, 2024

A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks
Nicholas Monath, Will Grathwohl, Michael Boratko, Rob Fergus, Andrew McCallum, Manzil Zaheer
Dense Retrieval New Perspective Dense Retriever Deep Encoder Stale Gradient Target Encoders Predictor Corrector Target Embeddings

August 14, 2024

Faster Stochastic Optimization with Arbitrary Delays via Asynchronous Mini-Batching
Amit Attia, Ofir Gaash, Tomer Koren
Stale Gradient Quantile Temporal Difference Arbitrary Delay Asynchronous Stochastic

May 7, 2024

FedStale: leveraging stale client updates in federated learning
Angelo Rodio, Giovanni Neglia
Federated Learning Algorithm Stale Gradient

March 22, 2024

Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddings
Yassaman Ebrahimzadeh Maboud, Muhammad Adnan, Divya Mahajan, Prashant J. Nair
Jina Embeddings Recommendation Model Stale Gradient

December 18, 2023

AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices
Ji Liu, Tianshi Che, Yang Zhou, Ruoming Jin, Huaiyu Dai, Dejing Dou, Patrick Valduriez
Model Training Decentralized Federated Learning Heterogeneous Device Stale Gradient Asynchronous Decentralized

December 16, 2023

Take History as a Mirror in Heterogeneous Federated Learning
Xiaorui Jiang, Hengwei Xu, Yu Gao, Yong Liao, Pengyuan Zhou
Federated Learning Heterogeneous Federated Learning Historical Text Asynchronous Federated Learning Etruscan Mirror Convergence Performance Stale Gradient

August 29, 2023

ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters
Xin Zhou, Ling Chen, Houming Wu
Stochastic Gradient Descent Stale Gradient Adaptive Batch Size Heterogeneous GPU Gradient Delay

August 25, 2023

Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Prediction
Guangji Bai, Ziyang Yu, Zheng Chai, Yue Cheng, Liang Zhao
Graph Neural Network Online Service Large Scale Graph GNN Training Large Scale Graph Datasets Dynamic Embeddings Stale Gradient GNN Training Framework

April 14, 2023

TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training
Tuo Zhang, Lei Gao, Sunwoo Lee, Mi Zhang, Salman Avestimehr
Asynchronous Federated Learning Cross Device Federated Learning Slow Convergence Stale Gradient Partial Model Training

February 2, 2023

Coordinating Distributed Example Orders for Provably Accelerated Training
A. Feder Cooper, Wentao Guo, Khiem Pham, Tiancheng Yuan, Charlie F. Ruan, Yucheng Lu, Christopher De Sa
Convergence Rate Context Example Order Matter Faster Training Stale Gradient

November 2, 2022

Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches
Haoze He, Parijat Dube
Stochastic Gradient Descent Mini Batch Synchronization Parameter Update Barrier Stale Gradient Asynchronous Algorithm Parallel Stochastic Multi Hop All Reduce

June 18, 2022

Pisces: Efficient Federated Learning via Guided Asynchronous Training
Zhifeng Jiang, Wei Wang, Baochun Li, Bo Li
Federated Learning Asynchronous Federated Learning Asynchronous Training Training Iteration Stale Gradient Participant Selection

May 31, 2022

Distributed Graph Neural Network Training with Periodic Stale Representation Synchronization
Zheng Chai, Guangji Bai, Liang Zhao, Yue Cheng
Graph Neural Network Graph Representation GNN Based Stale Gradient Subgraph Level Training GNN Training Framework

March 2, 2022

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data
Zihao Zhou, Yanan Li, Xuebin Ren, Shusen Yang
High Efficiency Raw Data Learning Rate Asynchronous Federated Learning Stale Gradient Staleness Problem Gradient Consistency

Stale Gradient

Papers

A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks

Faster Stochastic Optimization with Arbitrary Delays via Asynchronous Mini-Batching

FedStale: leveraging stale client updates in federated learning

Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddings

AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices

Take History as a Mirror in Heterogeneous Federated Learning

ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters

Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Prediction

TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training

Coordinating Distributed Example Orders for Provably Accelerated Training

Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches

Pisces: Efficient Federated Learning via Guided Asynchronous Training

Distributed Graph Neural Network Training with Periodic Stale Representation Synchronization

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data