Revisiting Gradient Staleness: Evaluating Distance Metrics for Asynchronous Federated Learning Aggregation

Patrick Wilhelm; Odej Kao

cs.LGMar 9, 2026

Revisiting Gradient Staleness: Evaluating Distance Metrics for Asynchronous Federated Learning Aggregation

Authors: Patrick Wilhelm, Odej Kao

Organizations: BIFOLD, Technische Universität Berlin, Berlin, Germany · Technische Universität Berlin, Berlin, Germany

Abstract

In asynchronous federated learning (FL), client devices send updates to a central server at varying times based on their computational speed, often using stale versions of the global model. This staleness can degrade the convergence and accuracy of the global model. Previous work, such as AsyncFedED, proposed an adaptive aggregation method using Euclidean distance to measure staleness. In this paper, we extend this approach by exploring alternative distance metrics to more accurately capture the effect of gradient staleness. We integrate these metrics into the aggregation process and evaluate their impact on convergence speed, model performance, and training stability under heterogeneous clients and non-IID data settings. Our results demonstrate that certain metrics lead to more robust and efficient asynchronous FL training, offering a stronger foundation for practical deployment.

Figures & tables

Staleness Metric	Low	Medium	High
Bregman	0.40 ± 0.02	0.40 ± 0.02	0.39 ± 0.01
Euclidean	0.41 ± 0.02	0.43 ± 0.01	0.44 ± 0.01
Fisher	0.34 ± 0.02	0.34 ± 0.01	0.35 ± 0.01
Manhattan	0.24 ± 0.00	0.24 ± 0.00	0.25 ± 0.02
Cosine	0.20 ± 0.02	0.27 ± 0.02	0.21 ± 0.03
KL-divergence	0.28 ± 0.03	0.17 ± 0.09	0.25 ± 0.05

TABLE I: Final Test Accuracy (Mean ± Std) for Each Staleness Metric Across Availability Scenarios [%]

Metric	Type	Rationale
L2 Distance	Euclidean norm	Measures magnitude of update deviation.
L1 Manhattan Distance	Euclidean norm (L1)	Measures total coordinate-wise deviation.
Cosine Distance	Directional similarity	Captures alignment/conflict in gradient directions.
Bregman Divergence	Information-theoretic (asymmetric)	Measures information loss using convex functions.
Hellinger Distance	Probabilistic (symmetric)	Useful for comparing probability distributions.
KL-Divergence	Information-theoretic (asymmetric)	Measures relative entropy

TABLE II: Selected distance/divergence metrics evaluated in our study

Scenario	Mean ( $\mu$ )	Std. Dev. ( $\sigma$ )	Max Delay (s)	Description
Low Asynchrony	1.0	0.5	3	Emulates environments with uniform devices and reliable network.
Moderate Asynchrony	3.0	1.0	6	Represents typical real-world heterogeneity in mobile and edge devices.
High Asynchrony	5.0	2.5	10	Simulates scenarios with frequent stragglers and inconsistent participation.

TABLE III: Asynchrony Scenarios used in Experiments

Fig. 1: Image classification using an CNN Model. Test accuracy over wall-clock time for the Low client heterogeneity scenario.

Fig. 2: Image classification using an CNN Model. Test accuracy over wall-clock time for the Medium client heterogeneity scenario.

Fig. 3: Image classification using an CNN Model. Test accuracy over wall-clock time for the High client heterogeneity scenario.

Staleness Metric	Low	Medium	High
Bregman	82.96 ± 1.35	83.57 ± 0.85	82.70 ± 0.78
Euclidean	82.86 ± 1.68	83.01 ± 1.10	81.90 ± 2.29
Fisher	82.22 ± 1.05	81.02 ± 1.61	82.21 ± 1.04
Manhattan	75.63 ± 4.93	71.77 ± 5.04	73.75 ± 8.41
Cosine	50.40 ± 27.69	65.25 ± 9.91	42.65 ± 28.15
KL-divergence	59.98 ± 16.01	37.19 ± 16.19	45.80 ± 26.85

TABLE IV: Final Test Accuracy (Mean ± Std) for Each Staleness Metric Across Availability Scenarios [%]

Fig. 4: Next Character Prediction using an LSTM Model. Test accuracy over wall-clock time for the Low client heterogeneity scenario.

Fig. 5: Next Character Prediction using an LSTM Model. Test accuracy over wall-clock time for the Medium client heterogeneity scenario.

Fig. 6: Next Character Prediction using an LSTM Model. Test accuracy over wall-clock time for the High client heterogeneity scenario.

Federated Learning Aggregation

Federated learning aggregation combines client updates into shared or personalized models while balancing data heterogeneity, robustness, privacy, and communication.

See all 260 papers

Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning
Nov 1, 2025Baris Askin, Holger R. Roth, Zhenyu Sun +3Asynchronous Federated Learning Federated Learning Aggregation
Carnegie Mellon University · NVIDIA · Northwestern University
FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learning
Feb 17, 2026Chaoyi Lu, Yiding Sun, Zhichuan Yang +3Asynchronous Federated Learning Federated Learning Aggregation
FedSteer: Taming Extreme Gradient Staleness in Federated Learning with Corrective Projections and Caching
Jun 8, 2026Haoran Zhang, Cainã Figueiredo Pereira, Marie Siew +3Federated Learning Aggregation Federated Learning
Electrical and Computer Engineering Dept., The University of Texas at Austin, Austin, Texas, USA · CERI/LIA, University of Avignon, Avignon, France · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore +2

Nov 1, 2025J/K move · Enter open · S save

Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

Baris Askin, Holger R. Roth, Zhenyu Sun, Carlee Joe-Wong+2

Carnegie Mellon University · NVIDIA · Northwestern University

Asynchronous Federated Learning Federated Learning Aggregation

Open paper AskPDF arXiv

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its scalability is limited by synchronization overhead. Asynchronous federated learning (AFL) alleviates this issue by allowing clients to communicate independently, thereby improving wall-clock efficiency in large-scale, hardware-heterogeneous environments. However, asynchrony introduces updates computed on outdated global models (staleness) that can destabilize optimization and hinder convergence. We propose FedRevive, an AFL framework that revives stale updates through data-free knowledge distillation (DFKD). FedRevive integrates parameter-space aggregation with a server-side DFKD process that transfers knowledge from stale client updates to the current global model without access to data. A meta-learned generator synthesizes pseudo-samples for multi-teacher distillation. A hybrid aggregation scheme combining raw with DFKD updates effectively mitigates staleness while retaining AFL scalability. Experiments on vision and text benchmarks show that FedRevive achieves faster training by up to 29.7% and higher final accuracy by up to 11.4 percentage points than baselines.

Abstract

Federated Learning Aggregation

Explore similar work

Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learning

FedSteer: Taming Extreme Gradient Staleness in Federated Learning with Corrective Projections and Caching