After the Federation: Benchmarking Knowledge Portability to Heterogeneous Newcomers

Xinyuan Zhao

cs.CVOct 19, 2026

After the Federation: Benchmarking Knowledge Portability to Heterogeneous Newcomers

Authors: Xinyuan Zhao

Organizations: Sun Yat-sen University

Abstract

Once historical clients become unavailable, a federation must serve newcomers through the artifact it leaves behind. How much of its knowledge remains useful in an architecture chosen afterward? We introduce FEDNBO, a seven-dataset benchmark that separates knowledge supply, policy identification, and private-model compilation under disjoint data roles. Controlled studies expose inflated oracle gains, architecture-dependent transfer reversals, and a gap between teacher accuracy and private-model utility. Ordinary public-response distillation provides a competitive reference without public task labels or newcomer samples: gains over strengthened local training are 23.52 and 16.62 percentage points on 300 CelebA identities and 300 FedEMNIST writers, respectively. In a complete-pipeline FedEMNIST comparison, this reference reaches 97.13% accuracy with 12.4 seconds of compilation per architecture, versus 97.03% and 67.5 seconds for a HeteroFedTTA reconstruction at the same epoch budget. Equal-count public-input interventions extend support sensitivity to trainable visual backbones. Together, these results establish an auditable benchmark and a strong reference for evaluating the utility, cost and reliability of post-federation knowledge transfer.

Figures & tables

Figure 1: The FedNBO protocol. (a) A frozen bank exports public inputs/responses. (b) Private architectures learn from unlabeled responses; disjoint response validation selects checkpoints. Ridge is a separate control. (c) Shadow policies and newcomer selection-label methods are alternative routes; local wins ties and query labels are evaluation-only. (d) Gain separates supply, identification and compilation. Target utilities are hidden at deployment; glyphs are schematic.

Dataset	classes	source/shadow/test	client unit	modalities	frozen bank $\rightarrow$ private models
CREMA-D [ 6 ]	6	60/15/16	actor (natural)	visual/audio features; some audio missing	global/cluster FedAvg MLPs $\rightarrow$ linear/three-layer MLP
MELD [ 22 ]	4	15/3/3	speaker (natural)	text/audio; dialogues remain disjoint	GloVe/audio MLPs $\rightarrow$ MobileBERT-feature linear/MLP
UCF51-20 [ 25 ]	51	12/4/4	group silo (simulated)	video/audio features	FedAvg MLPs $\rightarrow$ linear/MLP
TCGA-BRCA [ 27 ]	2	7/3/3	Tissue Source Site (natural)	WSI embedding/RNA expression	FedAvg MLPs $\rightarrow$ linear/MLP
UCI HAR [ 2 ]	6	12/6/12	subject (natural)	raw accelerometer/gyroscope windows	flattened-window MLPs $\rightarrow$ raw linear/temporal CNN
CelebA [ 16 ]	2	878/310/291; $+4{\times}300$ confirm	identity (natural)	raw RGB/facial landmarks	raw FedSGD ResNet-18 $\rightarrow$ MobileNetV3/EfficientNet-B0

Table 1: Benchmark tasks. TCGA-BRCA uses balanced accuracy; the others use accuracy. UCF51-20 clients are group-preserving simulations; all other client units are natural.

Figure 2: Task and supervision controls. A: Attribute gains on 300 decorrelated CelebA identities include the negative Necklace result. B: Response-minus-public-label-control gains in separate historical studies. These post-confirmation supervised controls are not compiler inputs. Bars: 95% client-bootstrap intervals; cohorts and local references differ across conditions.

Figure 3: Four onboarding audits. A: Same-query versus label-disjoint selection (95% newcomer intervals). B: Active Testing across label budgets. C: Mean selection and LCB power/harm under declared or shifted populations (50,000 trials). D: Fixed KD-1 gains on 12 HAR subjects, averaged over seeds within each architecture.

Dataset	1	2	3	4	Choice	Gain [95% CI]	Harm
	Fixed-policy gain				PCV deployment
CREMA-D	$-7.08$	$-6.78$	$-8.80$	$-6.87$	local	$0.00$ [0.00,0.00]	0.0
MELD	$+11.16$	$+16.05$	$+17.66$	$+17.89$	pure KD	$+17.89$ [15.30,22.03]	0.0
UCF51-20	$+1.90$	$+1.57$	$-1.18$	$-1.45$	local	$0.00$ [0.00,0.00]	0.0
TCGA-BRCA	$+13.33$	$+17.62$	$+17.18$	$+14.21$	joint KD 0.1	$+13.33$ [ $-1.31$ ,30.00]	33.3
UCI HAR	$+3.23$	$+3.88$	$+1.98$	$+1.69$	local	$0.00$ [0.00,0.00]	0.0

Table 3: Fixed policies and shadow-selected deployment. Gain/CI: pp; harm: %. Policies 1–4 are global/cluster 0/cluster 1/uniform on CREMA-D and KD 0.1/1/10/pure KD elsewhere. PCV uses only shadow labels; all LCB rules abstain. Test cohorts: MELD/TCGA, 3 clients; UCF, 4 simulated silos.

Policy	Gain [95% CI]	Harm	Linear gain	CNN gain
Fixed joint KD 1	$+3.88$ [2.19,5.51]	16.7	$-1.73$	$+9.49$
Disagreement audit, 6 labels	$+3.22$ [1.83,4.64]	8.3	$-1.08$	$+7.52$
Disagreement audit, 24 labels	$+4.47$ [2.71,6.34]	8.3	$-0.93$	$+9.87$
Same-query oracle (diagnostic)	$+8.03$	0.0	$+2.89$	$+13.18$

Table 4: HAR architectures, selection labels and compatible baselines. A: sealed 12-subject audit; gain/CI: pp, harm: %. Oracle is diagnostic. B: separate supported slices with different references; PeFLL uses 12 labeled descriptors, ATP an unlabeled batch of 20. Accuracy: %.

Figure 4: Public inputs and private transfer. On the same 270 shadow identities: A: anchor-count frontier; B: blur applied only to public inputs; C: broad-minus-restricted neural support with equal counts, 336 updates and 512 extra validation responses. Bars in C are paired 95% client intervals conditional on three draws; EfficientNet at 128 is inconclusive.

Private backbone	Fit inputs	Broad acc.	Restr. acc.	Difference [95% CI]	$\Delta$ KL	Harm
MobileNet blocks	64	72.07	69.01	3.06 [1.84,4.32]	$-0.0490$	27.4/32.2
	128	78.46	75.42	3.04 [1.93,4.20]	$-0.0426$	17.8/24.8
	512	85.32	80.94	4.38 [3.26,5.54]	$-0.0861$	10.4/13.7
EfficientNet blocks	64	75.96	70.12	5.83 [4.51,7.21]	$-0.0242$	25.6/35.2
	128	82.19	82.04	0.15 [ $-0.76$ ,1.10]	$+0.0248$	14.1/14.4
	512	87.82	84.94	2.89 [1.84,3.94]	$-0.0667$	8.5/13.0

Table 5: Equal-count public-support ablation. Broad/restricted pools share teacher, initialization, 336 updates, 512 validation responses and 270 identities. Accuracy/harm: %; differences: pp. CIs resample clients after averaging three draws.

\Delta

KL is broad minus restricted; harm is broad/restricted.

Architecture	Pipeline	Accuracy	Harm	Compile (s)	Target (ms)
MLP	Original + KD	95.87/95.82	12.0/12.0	11.8/23.1	0.71/0.66
	Meta + KD	95.74/95.97	12.7/11.0	11.0/22.4	0.66/0.66
	Meta + dual	95.76/95.82	12.3/12.0	67.1/132.2	6.11/6.23
CNN	Original + KD	98.39/98.33	2.0/2.3	12.9/25.7	0.79/0.77
	Meta + KD	98.44/98.49	2.3/2.7	12.8/25.3	0.69/0.69
	Meta + dual	98.29/98.42	2.7/2.0	67.9/136.5	42.88/10.94

Table 6: FedEMNIST compiler-budget sensitivity. Entries are 100/200 epochs with matched inputs, initialization, optimizer and batch order. Accuracy/harm: %; harm averages three seeds by architecture. Meta + dual reconstructs HeteroFedTTA. Timings exclude I/O/loading; source effort differs and is reported separately.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Study	roles	frozen choice	policy/result time	policy SHA
CelebA sparse handoff	878/310/291 identities	fit pseudo-label prototype; 8 labels	09-26 23:53/09-26 23:55	9f2392c1
CelebA ridge	new 300 identities	4,096 anchors; ridge alpha 1	09-27 01:26/09-27 01:27	59f616ab
CelebA trainable blocks	second new 300	late blocks; 8 epochs; response selection	09-27 02:07/09-27 02:09	561559fa
CelebA 3-label	third new 300	3 incidence labels; late blocks	09-27 10:39/09-27 10:40	a0b9c7e6
CelebA decorrelated 3-label	fourth new 300	correlation rule; inherited blocks	09-27 12:24/09-27 12:26	9a2de6ac
FedEMNIST	1000/400/300 writers	4,096 anchors; 2 architectures; 20-label local	09-27 13:53/09-27 14:02	6f66f2a4

Appendix

Table S1: Original confirmation chronology. Hash prefixes identify full policy hashes in the receipts. Revised baselines reuse these cohorts rather than create new confirmations.

Study (clients)	Local	Private	Teacher	Gain [95% CI]	Harm (%)
CelebA sparse (291)	63.49	65.23	92.40	1.74 [0.15,3.22]	37.5
CelebA ridge (300)	64.23	80.85	88.13	16.63 [14.63,18.63]	15.3
CelebA blocks (300)	63.15	86.46	88.17	23.31 [21.33,25.27]	6.0
CelebA correlated (300)	55.95	75.48	76.44	19.53 [17.92,21.19]	7.7
CelebA decorrelated (300)	56.40	66.07	67.19	9.66 [8.09,11.20]	19.7
FedEMNIST original (300)	77.90	96.75	99.34	18.85 [17.49,20.20]	1.3

Appendix

Table S2: Historical cohort endpoints, distinct from the strengthened references in main Table 2. Each row retains its own local baseline; rows are not paired interventions. Accuracy is %, gain is pp; the two attribute studies use balanced accuracy. Harm averages architectures; decorrelated ties use the recorded correction. The original FedEMNIST teacher was centralized.

Dataset	Four fixed gains (pp)	PCV choice	PCV gain [95% CI]	Harm
CREMA-D	-7.08 / -6.78 / -8.80 / -6.87	local	$0.00$ [0.00,0.00]	0.0%
MELD	+11.16 / +16.05 / +17.66 / +17.89	pure KD	$+17.89$ [15.30,22.03]	0.0%
UCF51-20	+1.90 / +1.57 / -1.18 / -1.45	local	$0.00$ [0.00,0.00]	0.0%
TCGA-BRCA	+13.33 / +17.62 / +17.18 / +14.21	joint KD 0.1	$+13.33$ [ $-1.31$ ,30.00]	33.3%
UCI HAR	+3.23 / +3.88 / +1.98 / +1.69	local	$0.00$ [0.00,0.00]	0.0%

Appendix

Table S3: Fixed-policy opportunity and deployable population selection. The four gains are global/cluster 0/cluster 1/uniform for CREMA-D and joint KD 0.1/1/10/pure KD elsewhere. PCV uses shadow labels only; local has zero gain. All LCB decisions abstain. TCGA and MELD have only three test clients.

Study/model	Fit	Broad acc.	Restr. acc.	Difference [95% CI]	$\Delta$ KL	Harm (%)
MobileNet blocks	64	72.07	69.01	3.06 [1.84,4.32]	-0.0490	27.4/32.2
MobileNet blocks	128	78.46	75.42	3.04 [1.93,4.20]	-0.0426	17.8/24.8
MobileNet blocks	512	85.32	80.94	4.38 [3.26,5.54]	-0.0861	10.4/13.7
EfficientNet blocks	64	75.96	70.12	5.83 [4.51,7.21]	-0.0242	25.6/35.2
EfficientNet blocks	128	82.19	82.04	0.15 [-0.76,1.10]	+0.0248	14.1/14.4
EfficientNet blocks	512	87.82	84.94	2.89 [1.84,3.94]	-0.0667	8.5/13.0

Appendix

Table S5: Equal-count support interventions. Accuracy is %; differences and intervals are pp. Broad/restricted conditions share teacher, clients and training budget within each study. Intervals resample clients after averaging three draws. Restricted means pose-tail on CelebA and four scripts on CREMA-D. Harm columns are broad/restricted; KL difference is broad minus restricted; lower KL is better. Studies use different representations and local references.

Study	Supply	Compile loss	Gain	Inherited	Compiler	Total
CelebA ridge	$+23.90$	$7.27$	$+16.63$	4.0%	11.3%	15.3%
CelebA blocks	$+25.02$	$1.71$	$+23.31$	4.0%	2.0%	6.0%
CelebA decorrelated	$+10.78$	$1.12$	$+9.66$	10.3%	9.3%	19.7%
FedEMNIST	$+21.44$	$2.59$	$+18.85$	0.3%	1.0%	1.3%

Appendix

Table S7: Historical harm decomposition using each cohort’s original local reference. Supply, loss and gain are pp; harm is a percentage after architecture averaging. All quantities use query labels and are diagnostic.

Figure S1 : Utility, candidate opportunity and private architecture. A: Strengthened local controls versus compilation on separate CelebA and FedEMNIST cohorts. B: Four fixed policies on each dataset. Policies 1–4 denote global, cluster 0, cluster 1 and uniform on CREMA-D; elsewhere they denote KD weights 0.1, 1, 10 and pure KD. These are dataset-specific candidate menus. C: UCI HAR gains by private architecture under fixed KD, 6- and 24-label disagreement audits, and a same-query diagnostic oracle. The oracle is not an allowed deployment method. Each comparison retains its original local reference.

Figure S2 : Complete-pipeline budget sensitivity. Accuracy, client harm, compilation time and target inference time at 100 and 200 compilation epochs. M/C denote pixel-MLP/CNN; Orig, Meta and Dual denote original+KD, meta+KD and meta+dual pipelines. Points are the reported three-seed means. Timing is sequential simulator time and excludes loading and I/O; it is not a parallel deployment latency estimate.

Figure S3 : Equal-count neural public-support comparison. Broad versus pose-restricted public inputs at 64, 128 and 512 fit images. M/E denote MobileNetV3-Small/EfficientNet-B0. Each setting uses 336 neural updates, 512 additional shared validation responses, three anchor draws and the same 270 shadow identities. Panels show accuracy and architecture-specific client harm; the main support figure reports paired accuracy differences with client-bootstrap intervals.

Method	unseen	frozen	multimodal	unseen arch.	private output
OD-PFL [ 1 ]	$\checkmark$	$\checkmark$	–	–	(predefined)
UPFL/HeteroFedTTA [ 28 ]	$\checkmark$	$\checkmark$	–	$\checkmark$	$\checkmark$
FLowDUP [ 29 ]	$\checkmark$	$\checkmark$	–	–	(predefined)
EPFL [ 15 ]	$\checkmark$	$\checkmark$	–	–	(predefined)
PeFLL [ 23 ]	$\checkmark$	$\checkmark$	–	–	(predefined)
pFedHN [ 24 ]	$\checkmark$	$\checkmark$	–	–	(predefined)

Appendix

Table S8: Novelty and applicability boundary.

Federated Knowledge Distillation

Federated knowledge distillation transfers predictive behavior among data-isolated clients to train useful models without sharing raw data.

See all 36 papers

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Sep 15, 2026Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro CavalcanteFederated Knowledge Distillation Non-IID Federated Learning
Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden
Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation
Jul 20, 2026Claudia Grosser, Maike Heuer, Denis Krompass +1Language Model Pretraining LLM Evaluation
Technical University Munich · Siemens AG
FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning
Sep 29, 2026Xinyuan ZhaoCross-Modal Knowledge Distillation Federated Knowledge Distillation
Sun Yat-sen University

Sep 15, 2026J/K move · Enter open · S save

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro Cavalcante

Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden

Federated Knowledge Distillation Non-IID Federated Learning Knowledge Distillation Federated Learning

Open paper AskPDF arXiv

Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head Adaptation (pFedKDH), which aggregates only the shared backbone, keeps persistent client-specific heads, and uses a recalibrated global head as a teacher during local training. Across MNIST, Fashion-MNIST, CIFAR10, and CIFAR100 under class-wise Dirichlet partitions, pFedKDH obtains the best accuracy in most settings, with accuracy gaps up to 37.67% over the weakest baseline and consistently low standard deviation across repetitions. Component-wise diagnostics and convergence results support the role of persistent heads and distillation-guided local optimization under label-skewed data.

Abstract

Federated Knowledge Distillation

Explore similar work

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation

FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning