Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

Baris Askin; Holger R. Roth; Zhenyu Sun; Carlee Joe-Wong; Gauri Joshi; Ziyue Xu

cs.LGNov 1, 2025

Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

Authors: Baris Askin, Holger R. Roth, Zhenyu Sun, Carlee Joe-Wong, Gauri Joshi, Ziyue Xu

Organizations: Carnegie Mellon University · NVIDIA · Northwestern University

Abstract

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its scalability is limited by synchronization overhead. Asynchronous federated learning (AFL) alleviates this issue by allowing clients to communicate independently, thereby improving wall-clock efficiency in large-scale, hardware-heterogeneous environments. However, asynchrony introduces updates computed on outdated global models (staleness) that can destabilize optimization and hinder convergence. We propose FedRevive, an AFL framework that revives stale updates through data-free knowledge distillation (DFKD). FedRevive integrates parameter-space aggregation with a server-side DFKD process that transfers knowledge from stale client updates to the current global model without access to data. A meta-learned generator synthesizes pseudo-samples for multi-teacher distillation. A hybrid aggregation scheme combining raw with DFKD updates effectively mitigates staleness while retaining AFL scalability. Experiments on vision and text benchmarks show that FedRevive achieves faster training by up to 29.7% and higher final accuracy by up to 11.4 percentage points than baselines.

Figures & tables

Figure 2: Test accuracy versus simulated time on CIFAR-10 , CIFAR-100 , 20NewsGroups , and FEMNIST . On CIFAR-10 , FedRevive converges faster and reaches higher final accuracy than the baselines. On the harder CIFAR-100 task, its advantage becomes more pronounced later in training, when client models provide stronger teachers. On 20NewsGroups , FedRevive converges faster than the baselines. On FEMNIST , asynchronous methods show similar overall convergence due to the relative simplicity of the task, but FedRevive attains among the highest final accuracy.

Method	CIFAR-10	CIFAR-100	20NewsGroups	FEMNIST
FedAvg	63.18 (631.1)	45.73 (>1000)	68.46 (51.2)	63.62 (>1000)
FedBuff	66.23 (96.7)	44.46 (320.7)	67.93 (13.5)	74.43 (38.0)
AFL-DW	53.62 (—)	37.89 (525.1)	67.40 (14.3)	75.07 (35.3)
FADAS	59.89 (214.7)	31.09 (—)	67.20 (18.1)	76.90 (146.0)
FedAC	59.71 (458.9)	43.39 (540.0)	67.14 (16.6)	78.34 (70.5)
FedFa	70.08 (90.3)	45.18 (318.3)	68.00 (11.1)	75.43 (44.8)

Table 1: Final test accuracy and time-to-target, reported as accuracy% (time), where time is the mean simulated time to reach 85% of FedBuff peak accuracy. Per dataset, the best two values are bold and the third best is underlined, separately for accuracy and time. FedRevive provides the strongest accuracy and speed on most datasets.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Average $D_{\mathrm{KL}}(\boldsymbol{\pi}_{i}\\|\hat{\boldsymbol{\pi}}_{i})$
CIFAR-10	0.4240 $\pm$ 0.0103
CIFAR-100	0.5539 $\pm$ 0.0174
20NewsGroups	0.5889 $\pm$ 0.0312
FEMNIST	0.4561 $\pm$ 0.0554

Appendix

Table 2: KL divergence between true client label proportions and the server-derived proxy proportions, reported as mean

\pm

std across seeds. We report the average client KL after the second received update.

Figure 5: FedRevive with estimated client label proportions on CIFAR-10 , CIFAR-100 , and FEMNIST . Replacing ground-truth client label proportions with our probing-based proxies yields similar training dynamics and final performance across datasets.

Dataset	Nb. of classes	Top-1 acc.	Top-2 acc.
CIFAR-10	10	$0.456\pm 0.022$	$0.673\pm 0.032$
CIFAR-100	100	$0.135\pm 0.001$	$0.219\pm 0.010$
20NewsGroups	20	$0.198\pm 0.015$	$0.276\pm 0.062$
FEMNIST	62	$0.175\pm 0.045$	$0.338\pm 0.014$

Appendix

Table 3: Top-1 and top-2 dominant-class recovery of the server-derived proxy proportions, reported as mean

\pm

std across seeds. Top-1 accuracy checks whether

\arg\max_{j}\hat{\pi}_{i,j}

matches the true dominant class of client

i

; top-2 accuracy checks whether the true dominant class appears in the proxy’s top-2 classes.

Dataset	Metric	Dirichlet concentration $\alpha$
		0.05	0.10	0.30	0.50	1.00	5.00
CIFAR-10	Top-1	0.930	0.899	0.822	0.741	0.691	0.437
	Top-2	0.995	0.989	0.960	0.915	0.872	0.611
CIFAR-100	Top-1	0.708	0.671	0.528	0.472	0.382	0.289
	Top-2	0.876	0.847	0.679	0.626	0.551	0.468

Appendix

Table 4: Dominant-class recovery across Dirichlet concentration levels after one local training round from random initialization. Entries are recovery rates across

1000

clients; smaller

\alpha

indicates stronger label skew.

Component	FLOPs (G) per iter †	Total across iters
Client local training (25 iters)	${1.65}$	$\mathbf{41.25}$
Generator train&synth. (every $T_{\text{gen}}$ rounds, amortized)	$0.14+0.11\,c$	$\mathbf{0.28+0.22\,c}$
Distillation (sequential implementation, 10 iters)	$1.65+0.55\,c$	$16.5+5.5\,c$
Distillation (effective cost w/ concurrent implement.)	$2.2$	$\mathbf{22}$

Appendix

Table 5: Compute overhead breakdown (FLOPs) per sample for CIFAR-10 experiments. FLOPs (G) for dominant components.

c

denotes the teacher buffer size. † Rows report total compute except the last row, which reports effective blocking compute with concurrent implementation (total FLOPs unchanged).

Figure 6: Distribution of update staleness across server rounds for a fully asynchronous setup. The distribution aligns with real-world AFL systems reported by Nguyen et al. [38] .

Figure 7: Wall-clock evaluation with server overhead on CIFAR-10 under different hardware assumptions. We adjust the CIFAR-10 timelines by adding measured and converted per-aggregation server runtimes for several server configurations (H200, H100, Tesla T4, CPU-32, and CPU-16), while leaving the underlying update sequence and training dynamics unchanged. With strong accelerators, the conclusions are essentially unchanged; with weaker servers, FedRevive and DD-FedRevive slow down in wall-clock time but still retain a clear accuracy advantage over the asynchronous baselines.

Figure 8: Staleness distribution induced by the modified delay schedule used in the experiment for robustness to change in delay scheme in Section 3.2 . For reference, the default distribution is shown in Fig. 6 .

Figure 9: Test accuracy over simulated time on CIFAR-10 , CIFAR-100 , and 20NewsGroups under milder heterogeneity (

\alpha=1

). FedRevive converges faster than FedBuff and achieves higher final accuracy across datasets.

Panel of Figure 10 — Figure 10: Raw accuracy curves across datasets ( $\alpha=0.5$ ).

Figure 11: Update contributions and the role of

\beta(\tau)

on CIFAR-10 . Left: Per-round effect of applying only the stale client update, only the KD-revived update, or their combination under the adaptive

\beta(\tau)

schedule. Right: Test accuracy vs. simulated time under adaptive and constant (

0.75

and

1

)

\beta(\tau)

. Constant schedules underweight the client update and degrade performance.

Panel of Figure 12 — Figure 12: CIFAR-10 update-component ablation under adaptive and fixed mixing-weight schedules.

Panel of Figure 13 — Figure 13: Comparison of different $\beta(\tau)$ functions. Training curves under different adaptive and constant mixing schedules on 20NewsGroups and FEMNIST . The method is generally robust to the choice of $\beta$ , while adaptive schedules provide the best overall trade-off by relying more on the raw client update at low staleness and shifting more weight to the KD-revived update as staleness increases.

Figure 14: Representative synthesized pseudo-samples used by FedRevive . The samples are generated on the server for data-free knowledge distillation and are not intended to be realistic reconstructions of private client data. In our setting, regular asynchronous FL updates appear to provide enough signal for knowledge transfer, but not enough to recover visually realistic client images.

Dataset	Method	$\eta_{c}$	$\eta_{s}$	Buffer / Window
CIFAR-10	FedAvg	0.0003	1.4	–
	FedBuff	0.0003	0.05	2
	FADAS	0.0003	0.004	2
	FedAC	0.0003	0.0001	2
	FedFa	0.0003	0.03	2
	AFL-DW	0.0003	0.10	–

Appendix

Table 6: Client and server learning rates used in each experiment, together with the buffer / window size for buffered async methods.

Dataset	KD learning rate	Distillation temperature
CIFAR-10	0.0003	2.0
CIFAR-100	0.0001	2.0
FEMNIST	0.0003	2.0
20NewsGroups	0.0001	2.0

Appendix

Table 7: Hyperparameters for DD-FedRevive . Shared:

10

distillation steps per server update, distillation batch size

32

, teacher buffer size

c=8

, and

\beta=0.5

(constant for DD-FedRevive ).

Dataset	KD LR	Generator LR	Latent noise LR	$\alpha_{\text{feature}}$
CIFAR-10	1e-4	0.003	0.001	0.003
CIFAR-100	3e-5	0.003	0.001	0.0003
FEMNIST	1e-4	0.01	0.003	0.3
20NewsGroups	3e-4	0.01	0.1	0.0001

Appendix

Table 8: Dataset-specific DFKD hyperparameters for FedRevive . Shared: synthesis batch size

64

, KD batch size

32

, teacher buffer size

8

, KD temperature

\text{T}=1

2

generator steps every

10

server updates, and

10

KD steps per server update. Learning rates (LR) are presented in the table.

Figure 15: Different candidate

\beta

scheduling functions controlling the mix between parameter aggregation and DFKD updates.

1-\mathrm{cosine}

schedule is chosen, as it empirically shows the best performance on FEMNIST and 20NewsGroups .

Asynchronous Federated Learning

Asynchronous federated learning lets distributed clients train shared models without waiting for every participant, despite delays, churn, and uneven data.

See all 24 papers

FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learning
Feb 17, 2026Chaoyi Lu, Yiding Sun, Zhichuan Yang +3Asynchronous Federated Learning Federated Learning Aggregation
DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning
Mar 19, 2026Yizhou Han, Di Wu, Blesson VargheseFederated Learning Federated Continual Learning
School of Computer Science, University of St Andrews, UK
Revisiting Gradient Staleness: Evaluating Distance Metrics for Asynchronous Federated Learning Aggregation
Mar 9, 2026Patrick Wilhelm, Odej KaoFederated Learning Aggregation Asynchronous Federated Learning
BIFOLD, Technische Universität Berlin, Berlin, Germany · Technische Universität Berlin, Berlin, Germany

Feb 17, 2026J/K move · Enter open · S save

FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learning

Chaoyi Lu, Yiding Sun, Zhichuan Yang, Jinqian Chen+2

Asynchronous Federated Learning Federated Learning Aggregation

Open paper AskPDF arXiv

Asynchronous Federated Learning (AFL) has emerged as a significant research area in recent years. By not waiting for slower clients and executing the training process concurrently, it achieves faster training speed compared to traditional federated learning. However, due to the staleness introduced by the asynchronous process, its performance may degrade in some scenarios. Existing methods often use the round difference between the current model and the global model as the sole measure of staleness, which is coarse-grained and lacks observation of the model itself, thereby limiting the performance ceiling of asynchronous methods. In this paper, we propose FedPSA (Parameter Sensitivity-based Asynchronous Federated Learning), a more fine-grained AFL framework that leverages parameter sensitivity to measure model obsolescence and establishes a dynamic momentum queue to assess the current training phase in real time, thereby adjusting the tolerance for outdated information dynamically. Extensive experiments on multiple datasets and comparisons with various methods demonstrate the superior performance of FedPSA, achieving up to 6.37% improvement over baseline methods and 1.93% over the current state-of-the-art method.

Abstract

Asynchronous Federated Learning

Explore similar work

FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learning

DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning

Revisiting Gradient Staleness: Evaluating Distance Metrics for Asynchronous Federated Learning Aggregation