DROD: Dual-Ratio Ordered Dropout for Federated Continual Learning on Heterogeneous Devices

Berk Sezer; Özgür Soysal

cs.LGOct 18, 2026

DROD: Dual-Ratio Ordered Dropout for Federated Continual Learning on Heterogeneous Devices

Authors: Berk Sezer, Özgür Soysal

Organizations: Middle East Technical University · Stanford University

Abstract

Federated learning for capacity-heterogeneous clients and federated continual learning are each well studied on their own, but their intersection -- clients holding differently sized models while every client's data stream also drifts over time -- remains underexplored. We study this setting directly, training a ResNet-18 fleet from scratch on Split-CIFAR-100 (10 tasks, class-incremental), and introduce DROD (Dual-Ratio Ordered Dropout), which builds on FjORD's Ordered Dropout by using two independent reduction axes to extract sub-models: a shortcut ratio and a branch ratio. On a shared parameter/FLOP budget ladder, DROD achieves the highest mean accuracy among the heterogeneous-model federated learning strategies we compare. Its advantage over the two strongest baseline geometries persists across every replay-buffer size we test and under ER, DER++, and BiC; under EWC, all three geometries collapse alike.

Figures & tables

Panel of Figure 1 — Figure 1: One residual block under three configurations, $C{=}64$ ; in every panel each tensor is drawn at its full capacity, with the channels that tier keeps in color and the dropped remainder in gray, and each label reading kept/total. (a) The full, unreduced block, shown as a baseline; (b) width slicing / FjORD, where the shortcut and the branch narrow together to the same width; (c) DROD’s interior, where the shortcut stays near-full-width ( $s{=}0.70$ ) and only the branch narrows to a low rank ( $b{=}0.49$ ) – a low-rank transform on a near-full-width shortcut, decoupled rather than welded. Shortcut-tiering, the precursor technique that pushes $b\to 0$ , needs no diagram of its own: it collapses the branch to $F(x){=}0$ , leaving the shortcut passed through unchanged.

reduction $\times$ aggregation	ACC	forg.	ACC	forg.	params	FLOPs
	class-IL		Task-IL		% saved, mid / low
DROD $\theta{=}0.50$ $\dagger$	$0.3125\pm 0.0160$	$0.251\pm 0.019$	$0.6973\pm 0.0204$	$0.003\pm 0.004$	74.9 / 93.6	74.8 / 93.7
DROD $\theta{=}1.00$ $\dagger$	$0.2812\pm 0.0048$	$0.276\pm 0.008$	$0.6663\pm 0.0011$	$0.012\pm 0.004$	75.0 / 93.5	75.4 / 94.0
DROD $\theta{=}0.25$ $\dagger$	$0.2760\pm 0.0091$	$0.277\pm 0.010$	$0.6587\pm 0.0103$	$0.014\pm 0.006$	74.8 / 93.7	74.8 / 93.7
Width $\times$ FjORD $\dagger$	$0.2741\pm 0.0084$	$0.263\pm 0.004$	$0.6617\pm 0.0111$	$0.006\pm 0.004$	74.9 / 93.6	74.9 / 93.7
DROD $\theta{=}0.75$ $\dagger$	$0.2723\pm 0.0083$	$0.283\pm 0.013$	$0.6594\pm 0.0077$	$0.012\pm 0.008$	74.7 / 93.6	74.7 / 93.8

Table 1: Reduction axis and aggregation rule at a matched inference-FLOP budget on Split-CIFAR-100 with a fleet-wide replay buffer of 6,400. Results are mean

\pm

sample sd over three seeds; accuracies are fleet means across capacity tiers. The last two columns report mid/low-tier parameter and FLOP savings relative to the full model. † The parameter and FLOP budgets coincide. ‡ ScaleFL uses its native width/depth split and is not budget matched.

CL objective	DROD $\theta{=}0.50$	Width $\times$ FjORD	Shortcut $\times$ FjORD
ER	$\mathbf{0.3125}\pm 0.0160$	$0.2741\pm 0.0084$	$0.2518\pm 0.0143$
DER++	$\mathbf{0.2958}\pm 0.0008$	$0.2745\pm 0.0087$	$0.2506\pm 0.0098$
BiC	$\mathbf{0.2308}\pm 0.0076$	$0.2047\pm 0.0115$	$0.2003\pm 0.0070$
EWC	$0.0645\pm 0.0010$	$0.0609\pm 0.0020$	$0.0621\pm 0.0007$

Table 2: Generality across continual-learning methods. Fleet-mean class-IL ACC (mean

\pm

sample sd over

n{=}3

seeds) for the three leading fleet geometries, each trained under four CL objectives on an identical replay buffer (6,400, uniform allocation) and an identical budget. Columns are reduction geometries and rows are objectives. DROD is highest in every row; bold marks that lead, and the EWC row carries none because all three fleets collapse there and the

0.004

separating them is negligible. Per-tier values, forgetting and Task-IL are in Table 8 .

Figure 2: Fleet-mean class-IL accuracy against replay-buffer total, uniform per-tier allocation (every tier gets the same buffer);

n{=}3

, error bars

\pm 1

sample sd. DROD is above both baselines at every buffer total. Per-cell values: Table 9 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	setting	value
Data	dataset	CIFAR-100, 10 tasks $\times$ 10 classes
	class order	fixed (identity), not shuffled across seeds
	augmentation	random crop $32$ , pad $4$ (black fill) $+$ horizontal flip
	batch size	32 (train), 512 (eval)
	client partition	Dirichlet $\alpha{=}0.5$ , redrawn per task FL
Model	architecture	ResNet-18, CIFAR stem, from scratch

Appendix

Table 3: Complete experimental configuration. “FL” marks a value that differs from the centralized reference recipe because of the federated structure itself, not as a tuning choice.

cell	fleet class-IL ACC	high (t100)	mid (t025)	low (t006)
DROD $\theta$ =0.50 †	0.3125 $\pm$ 0.0160	0.3076	0.3161	0.3137
DROD $\theta$ =1.00 †	0.2812 $\pm$ 0.0048	0.2795	0.2840	0.2801
DROD $\theta$ =0.25 †	0.2760 $\pm$ 0.0091	0.2709	0.2785	0.2784
Width $\times$ FjORD (kd=false) †	0.2741 $\pm$ 0.0084	0.2701	0.2815	0.2708
DROD $\theta$ =0.75 †	0.2723 $\pm$ 0.0083	0.2680	0.2766	0.2723
Width $\times$ FjORD (kd=true) †	0.2719 $\pm$ 0.0157	0.2493	0.2896	0.2770

Appendix

Table 4: Per-tier class-IL ACC on the shared parameter ladder, buf=6,400,

n{=}3

, mean

\pm

sample sd, sorted by fleet mean. The high/mid/low columns give the per-tier split behind each fleet mean (tier keys t100/t025/t006); the fleet column is their unweighted mean. This is the params-matched companion to the FLOPs-matched main grid (Table 1 ), which reports the same fleet means without the per-tier breakdown. The DROD rows span the

\theta

axis:

\theta{=}1.00

is the full-width-shortcut endpoint,

\theta{\in}\{0.25,0.50,0.75\}

interior; the width-slicing endpoint

\theta{=}0

is the Width

\times

FjORD (kd=false) row itself. † sits on both ladders – narrowing channels cuts parameters and inference FLOPs together, so one run meets both budgets. ‡ ScaleFL keeps its own combined width/depth split and sits on neither ladder.

cell	fleet class-IL ACC	high (t100)	mid (t025)	low (t006)
DROD $\theta$ =0.50 †	0.3125 $\pm$ 0.0160	0.3076	0.3161	0.3137
DROD $\theta$ =1.00 †	0.2812 $\pm$ 0.0048	0.2795	0.2840	0.2801
DROD $\theta$ =0.25 †	0.2760 $\pm$ 0.0091	0.2709	0.2785	0.2784
Width $\times$ FjORD (kd=false) †	0.2741 $\pm$ 0.0084	0.2701	0.2815	0.2708
DROD $\theta$ =0.75 †	0.2723 $\pm$ 0.0083	0.2680	0.2766	0.2723
Width $\times$ FjORD (kd=true) †	0.2719 $\pm$ 0.0157	0.2493	0.2896	0.2770

Appendix

Table 5: Same grid on the shared inference-FLOP ladder, buf=6,400. The

\dagger

rows (width-family and DROD) are the same runs as on the parameter ladder – parameters and FLOPs move together for those axes (Section 2 ) – and are repeated here so the ranking is complete; the depth and shortcut rows are re-solved to hit the FLOP target and so differ from Table 4 . Depth

\times

FIARSE has no cell on this ladder (a magnitude mask removes no multiplies; see Table 1 ). ‡ ScaleFL sits on neither ladder.

cell	fleet class-IL ACC	high (t100)	mid (t025)	low (t006)
DROD $\theta$ =0.50 †	0.0863 $\pm$ 0.0044	0.0832	0.0870	0.0886
DROD $\theta$ =0.25 †	0.0845 $\pm$ 0.0032	0.0828	0.0848	0.0861
DROD $\theta$ =1.00 †	0.0845 $\pm$ 0.0002	0.0833	0.0849	0.0854
DROD $\theta$ =0.75 †	0.0822 $\pm$ 0.0029	0.0806	0.0832	0.0829
Width $\times$ FjORD (kd=true) †	0.0818 $\pm$ 0.0036	0.0790	0.0819	0.0845
Shortcut $\times$ FIARSE	0.0798 $\pm$ 0.0016	0.0804	0.0810	0.0779

Appendix

Table 6: Same grid, parameter ladder, buf=200. Columns, sorting and the

\dagger

\ddagger

markers as in Table 4 .

cell	fleet class-IL ACC	high (t100)	mid (t025)	low (t006)
DROD $\theta$ =0.50 †	0.0863 $\pm$ 0.0044	0.0832	0.0870	0.0886
DROD $\theta$ =0.25 †	0.0845 $\pm$ 0.0032	0.0828	0.0848	0.0861
DROD $\theta$ =1.00 †	0.0845 $\pm$ 0.0002	0.0833	0.0849	0.0854
DROD $\theta$ =0.75 †	0.0822 $\pm$ 0.0029	0.0806	0.0832	0.0829
Width $\times$ FjORD (kd=true) †	0.0818 $\pm$ 0.0036	0.0790	0.0819	0.0845
Width $\times$ FjORD (kd=false) †	0.0797 $\pm$ 0.0001	0.0778	0.0794	0.0819

Appendix

Table 7: Same grid, inference-FLOP ladder, buf=200. Columns, sorting and the

\dagger

\ddagger

markers as in Table 4 ; Depth

\times

FIARSE again has no cell on this ladder.

CL objective	fleet	class-IL ACC	high	mid	low	class-IL forg.
ER	DROD	$0.3125\pm 0.0160$	0.3076	0.3161	0.3137	0.251
	Width	$0.2741\pm 0.0084$	0.2701	0.2815	0.2708	0.263
	Shortcut	$0.2518\pm 0.0143$	0.2530	0.2459	0.2564	0.291
DER++	DROD	$0.2958\pm 0.0008$	0.2947	0.2975	0.2952	0.322
	Width	$0.2745\pm 0.0087$	0.2805	0.2835	0.2596	0.326
	Shortcut	$0.2506\pm 0.0098$	0.2460	0.2501	0.2555	0.324

Appendix

Table 8: Generality across CL methods, full per-tier breakdown (companion to Table 2 ). Buffer 6,400, uniform allocation,

n{=}3

, mean

\pm

sample sd. Tiers are the shared ladder’s high, mid and low rungs. In every one of the twelve method

\times

tier comparisons below, DROD’s tier is above the corresponding tier of both baselines.

buffer	DROD $\theta{=}0.50$	Width $\times$ FjORD	Shortcut $\times$ FjORD
3,200	$0.2414\pm 0.0102$	$0.2140\pm 0.0094$	$0.1918\pm 0.0051$
6,400	$0.3125\pm 0.0160$	$0.2741\pm 0.0084$	$0.2518\pm 0.0143$
12,800	$0.3547\pm 0.0186$	$0.3333\pm 0.0204$	$0.3025\pm 0.0087$

Appendix

Table 9: Fleet-mean class-IL ACC against replay-buffer total, uniform per-tier allocation (every tier gets the same buffer). Plain ER,

n{=}3

, mean

\pm

sample sd. DROD is highest at all three buffer totals. Per-tier values: Table 10 .

fleet	buffer	fleet	high	mid	low
DROD $\theta{=}0.50$	3,200	$0.2414\pm 0.0102$	0.2341	0.2423	0.2478
	6,400	$0.3125\pm 0.0160$	0.3076	0.3161	0.3137
	12,800	$0.3547\pm 0.0186$	0.3516	0.3603	0.3523
Width $\times$ FjORD	3,200	$0.2140\pm 0.0094$	0.2080	0.2170	0.2170
	6,400	$0.2741\pm 0.0084$	0.2701	0.2815	0.2708
	12,800	$0.3333\pm 0.0204$	0.3354	0.3472	0.3175

Appendix

Table 10: Fleet-mean class-IL ACC against replay-buffer total, uniform per-tier allocation (every tier gets the same buffer). Plain ER,

n{=}3

, mean

\pm

sample sd. Per-tier breakdown, one block per fleet; the fleet column repeats Table 9 . DROD’s high, mid and low tier are each above the corresponding tier of both baselines at all three buffer totals.

Figure 3: The

(s,b)

iso-budget curve

bs^{2}=\rho

for our mid tier (

\rho{=}0.2514

), with the two literature baselines marked explicitly. FjORD / width slicing sits at

\theta{=}0

, the single point

(s{=}0.5,\,b{=}1)

where the shortcut and branch narrow together; shortcut-tiering’s

b\to 0

precursor is drawn as the

b{=}0

axis itself, reachable at any

s

but off this particular curve, since it is a discrete, per-block toggle rather than a point on a nonzero-

\rho

curve. DROD spans the full nonzero-budget curve from the FjORD endpoint at

\theta{=}0

to the full-shortcut, low-rank endpoint at

\theta{=}1

; shortcut-tiering is shown separately as the limiting

b{=}0

axis.

Federated Continual Learning

Federated continual learning enables distributed clients to adapt models to changing data while retaining past knowledge and keeping raw data local.

See all 37 papers

DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning
Mar 19, 2026Yizhou Han, Di Wu, Blesson VargheseFederated Learning Federated Continual Learning
School of Computer Science, University of St Andrews, UK
Accurate and Resource-Efficient Federated Continual Learning
Jun 9, 2026Jebacyril Arockiaraj, Dhruv Parikh, Jayashree Adivarahan +2Communication-Efficient Distributed Training Continual Learning
University of Southern California · Los Angeles, USA · DEVCOM Army Research Office +1
ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning
Sep 30, 2026Sungmin Kang, Zhengzhong Tu, Sunwoo LeeFederated Continual Learning Replay-Based Continual Learning
Texas A&M University · Inha University

Mar 19, 2026J/K move · Enter open · S save

DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning

Yizhou Han, Di Wu, Blesson Varghese

School of Computer Science, University of St Andrews, UK

Federated Learning Federated Continual Learning Asynchronous Federated Learning Clustered Federated Learning

Open paper AskPDF arXiv

In real-world Federated Learning (FL) deployments, data distributions on devices that participate in training evolve over time. This leads to asynchronous data drift, where different devices shift at different times and toward different distributions. Mitigating such drift is challenging: frequent retraining incurs high computational cost on resource-constrained devices, while infrequent retraining degrades performance on drifting devices. We propose DriftGuard, a federated continual learning framework that efficiently adapts to asynchronous data drift. DriftGuard adopts a Mixture-of-Experts (MoE) inspired architecture that separates shared parameters, which capture globally transferable knowledge, from local parameters that adapt to group-specific distributions. This design enables two complementary retraining strategies: (i) global retraining, which updates the shared parameters when system-wide drift is identified, and (ii) group retraining, which selectively updates local parameters for clusters of devices identified via MoE gating patterns, without sharing raw data. Experiments across multiple datasets and models show that DriftGuard matches or exceeds state-of-the-art accuracy while reducing total retraining cost by up to 83%. As a result, it achieves the highest accuracy per unit retraining cost, improving over the strongest baseline by up to 2.3x. DriftGuard is available for download from https://github.com/blessonvar/DriftGuard.

Abstract

Federated Continual Learning

Explore similar work

DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning

Accurate and Resource-Efficient Federated Continual Learning

ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning