Rethinking Model Aggregation for Federated Learning with Domain Skew: A Knowledge-Preserving Perspective

Haotian Wu; Xu Ji; Qiheng Li; Li Xiong; Zibin Pan

cs.DCNov 1, 2026

Rethinking Model Aggregation for Federated Learning with Domain Skew: A Knowledge-Preserving Perspective

Authors: Haotian Wu, Xu Ji, Qiheng Li, Li Xiong, Zibin Pan

Organizations: Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou 510006, China · Guangdong Key Laboratory of Industrial Control System Security, Guangzhou 510555, China

Abstract

Domain skew challenges federated learning by requiring a shared model to accommodate distinct visual distributions. Domain-specific experts can capture these distributions separately, creating an opportunity to integrate their representations in a unified predictor. We propose Federated Knowledge-Preserving Learning (FKPL), a two-stage framework that combines domain-oriented expert training with prototype-based classifier calibration. First, a one-shot subspace-based clustering procedure groups clients for intra-cluster training. The resulting expert encoders are concatenated and frozen, preserving each learned feature mapping during global integration. Second, clients interpolate their concatenated features with class-matched domain prototypes and upload the resulting virtual features. The server trains a global linear classifier on these features, concentrating classifier optimization at the server after a single feature upload. FKPL achieves average accuracies of 73.01%, 67.00%, and 64.51% on Office-Caltech-10, PACS, and a four-domain subset of DomainNet, respectively. On DomainNet, its cross-domain standard deviation is 7.50 percentage points, the lowest among the evaluated methods. With the same concatenated extractor, prototype-based calibration improves average accuracy over federated classifier training by 9.99 and 14.94 percentage points on Office-Caltech-10 and PACS. These results identify classifier calibration as an effective way to combine preserved domain representations into a global predictor.

Figures & tables

Figure 1: Federated learning across multiple domains. The reported domain accuracies compare FedAvg with domain-specific independent training (SOLO), illustrating the performance gap that motivates preserving specialized representations.

Method	Client or feature organization	Integrated predictor	Classifier training
FedAvg [ 25 ]	Shared parameters	One shared encoder and head	Federated parameter averaging
PACFL [ 30 ]	One-shot subspace-based clusters	Cluster models	Within-cluster training
FPL [ 16 ]	Prototype-guided representation learning	Shared global model	Federated model optimization
FedConcat [ 5 ]	Label-distribution-based clusters	Concatenated frozen encoders and one head	Federated training of the global head
FKPL	Subspace-based clusters and class-matched domain prototypes	Concatenated frozen encoders and one head	Server-side training on uploaded virtual features

Table 1: Representative mechanisms for combining heterogeneous client information. Entries describe the cited methods; prototype use and clustering can also be combined with other FL designs.

Figure 2: t-SNE visualization of PACS representations. The top row shows independently trained domain models; the bottom row shows the shared FedAvg model. The differences in class separation motivate retaining domain-specific encoders when constructing a unified predictor.

Panel of Figure 3 — Figure 3: The FKPL pipeline. All local prototypes, domain prototypes, and virtual features use the same $dV$ -dimensional concatenated representation. Raw images are encoded locally. After virtual-feature collection, classifier optimization is performed at the server with the encoders frozen.

Method	Caltech	Amazon	Webcam	DSLR	AVG $\uparrow$	STD $\downarrow$
FedAvg	69.02	79.79	83.10	44.67	69.15	17.39
FedProx	69.02	78.63	73.79	46.67	67.03	14.13
MOON	68.13	79.16	79.31	45.33	67.98	15.98
FedProc	66.16	77.79	71.38	40.67	64.00	16.26
FedProto	69.11	79.68	84.48	57.33	72.65	12.06
FPL	73.57	81.16	82.76	48.00	71.37	16.09

Table 2: Accuracy (%) on Office-Caltech-10. AVG and STD are the mean and sample standard deviation across the listed domain accuracies. In the summary columns, boldface denotes the best value overall and underlining the best comparison value. The last two rows are architecture controls.

Method	Photo	Art painting	Cartoon	Sketch	AVG $\uparrow$	STD $\downarrow$
FedAvg	64.37	49.85	69.40	50.55	58.54	9.85
FedProx	64.01	49.36	70.09	50.75	58.55	10.14
MOON	65.63	47.25	70.43	53.50	59.20	10.69
FedProc	71.80	50.93	77.18	64.61	66.13	11.37
FedProto	64.61	45.15	68.08	48.97	56.70	11.33
FPL	66.47	51.33	72.91	56.69	61.85	9.68

Table 3: Accuracy (%) on PACS. AVG and STD are the mean and sample standard deviation across the listed domain accuracies. In the summary columns, boldface denotes the best value overall and underlining the best comparison value. The last two rows are architecture controls.

Method	Clipart	Painting	Real	Sketch	AVG $\uparrow$	STD $\downarrow$
FedAvg	66.52	59.99	75.13	50.98	63.16	10.21
FedProx	67.58	59.64	75.04	51.52	63.45	10.14
MOON	66.17	58.85	76.13	50.67	62.96	10.83
FedProc	67.40	60.82	74.77	50.96	63.49	10.11
FedProto	66.23	59.47	74.53	50.51	62.69	10.19
FPL	68.03	59.55	74.49	50.72	63.20	10.33

Table 4: Accuracy (%) on Domain-Net4. AVG and STD are the mean and sample standard deviation across the listed domain accuracies. In the summary columns, boldface denotes the best value overall and underlining the best comparison value. The last two rows are architecture controls.

Panel of Figure 4 — Figure 4: Sensitivity to the mixing bounds $a$ and $b$ in Eq. ( 10 ). Each cell gives AVG with cross-domain STD in parentheses.

	Office-Caltech-10
Methods	Caltech	Amazon	Webcam	DSLR	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$
Federated head	63.39	77.89	74.14	36.67	63.02	18.61
PCC (proposed)	63.84	77.37	74.14	76.67	73.01	6.27

	PACS
Methods	Photo	Art painting	Cartoon	Sketch	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$

Table 5: Ablation of classifier training with the DETFA concatenated extractor fixed.

	Office-Caltech-10
Methods	Caltech	Amazon	Webcam	DSLR	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$
FedAvg encoder	70.98	80.00	75.86	56.67	70.88	10.16
DETFA (proposed)	63.84	77.37	74.14	76.67	73.01	6.27

	PACS
Methods	Photo	Art painting	Cartoon	Sketch	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$

Table 6: Ablation of encoder construction with PCC used for classifier calibration.

Panel of Figure 5 — Figure 5: t-SNE visualizations of FKPL features on Office-Caltech-10 and PACS. Each panel shows features from the participating domains.

Panel of Figure 6 — Figure 6: Accuracy of the cluster-specific classifiers trained on the frozen FKPL representation.

	Office-Caltech-10
Representation	PSNR $\downarrow$	SSIM $\downarrow$	Feature Loss
Raw feature	8.43 $\pm$ 2.21	0.2073 $\pm$ 0.0368	0.0543 $\pm$ 0.0392
Local prototype	9.52 $\pm$ 2.09	0.2087 $\pm$ 0.0436	0.0444 $\pm$ 0.0237
Aggregated prototype	9.19 $\pm$ 0.60	0.1973 $\pm$ 0.0187	0.0410 $\pm$ 0.0100
Virtual feature	6.92 $\pm$ 1.39	0.1685 $\pm$ 0.0243	0.0481 $\pm$ 0.0237

Table 7: Representation-inversion results on Office-Caltech-10 and PACS. PSNR is measured in dB; lower PSNR and SSIM indicate lower reconstruction similarity under the evaluated attack. Feature loss measures the representation-matching objective.

	Office-Caltech-10
Representation	PSNR $\downarrow$	SSIM $\downarrow$	Feature Loss
Shared cluster prototype	9.56 $\pm$ 2.09	0.2154 $\pm$ 0.0389	0.0397 $\pm$ 0.0207
Leave-one-out others prototype	8.62 $\pm$ 1.84	0.2013 $\pm$ 0.0308	0.0532 $\pm$ 0.0280

	PACS
Representation	PSNR $\downarrow$	SSIM $\downarrow$	Feature Loss

Table 8: Representation inversion by an adversarial in-cluster client. The derived target is the mean prototype of the other contributing clients in Eq. ( 15 ).

Protocol component	Scalar count over all clients
Subspace-summary uploads	$d_{x}\sum_{m=1}^{M}p_{m}$
Expert-model uploads and downloads	$2TMP_{w}$
Concatenated-encoder downloads	$MVP_{f}$
Local-prototype uploads	$dV\sum_{m=1}^{M}\|\mathcal{K}_{m}\|$
Domain-prototype downloads	$MdV\sum_{k=1}^{K}\|\mathcal{V}_{k}\|$
Virtual-feature uploads	$N_{g}dV$

Table 9: Analytical communication counts in transmitted scalar values. Expert training includes

T

rounds; all remaining entries are one-time exchanges. Labels and message metadata are excluded. Each client receives the full concatenated encoder and complete available domain-prototype collection.

Recorded operation	Office-Caltech-10	PACS
	10 clients	20 clients
One-shot preprocessing time (s)	0.436	7.309
Server hierarchical-clustering time (s)	0.006	0.015
Stage I: total runtime (s)	766.930	8,024.443
Stage I: mean runtime per round (s)	7.669	80.244
Stage II: server calibration time (s)	4.424	21.437

Table 10: Recorded runtime at the classification client scales. Stage I uses 100 communication rounds and Stage II uses 200 classifier-calibration epochs. Server calibration time covers optimization of the linear head.

Metric	Office-Caltech-10	PACS
	10 clients	20 clients
Peak GPU memory: encoder stage (MB)	172.825	276.365
Peak GPU memory: classifier stage (MB)	133.022	103.481
Peak GPU memory: evaluation (MB)	329.966	623.595
Mean encoder-round energy (J)	180.680	3,082.015
One-time classifier-stage energy (J)	149.857	520.764

Table 11: Recorded memory and energy at the classification client scales. Encoder energy is measured per expert-training round; classifier-stage energy is measured for its one-time execution.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Office-Caltech-10	PACS	Domain-Net4
Federated training	Communication rounds $T$	100	100	100
	Local epochs $E$	10	10	10
	Mini-batch size $B$	64	64	64
	Optimizer	SGD	SGD	SGD
	Learning rate $\eta$	0.01	0.01	0.01
	Momentum	0.9	0.9	0.9

Appendix

Table A1: Federated training, model architecture, and client allocation.

Category	Hyperparameter	Office-Caltech-10	PACS	Domain-Net4
Client clustering	Subspace extraction	Truncated SVD	Truncated SVD	Truncated SVD
	Client-level basis-vector budget	10	10	10
	Similarity metric	Minimum principal angle $\theta_{1}$	Minimum principal angle $\theta_{1}$	Minimum principal angle $\theta_{1}$
	Clustering algorithm	Agglomerative hierarchical clustering	Agglomerative hierarchical clustering	Agglomerative hierarchical clustering
	Linkage criterion	Average linkage	Average linkage	Average linkage
	Clustering threshold (degrees)	5.3	4.2	11.5

Appendix

Table A2: Client clustering, prototype-based classifier calibration, and personalization.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 992 papers

Domain-Skewed Federated Learning with Feature Decoupling and Calibration
Mar 15, 2026Huan Wang, Jun Shen, Jun Yan +1Federated Learning Heterogeneous Federated Learning
School of Computing and Information Technology, University of Wollongong, Australia · School of Computing and Information Systems, Singapore Management University, Singapore
Joint Domain-Class Modeling for Federated Learning Under Feature Skew
Sep 19, 2026Sina Najafi, Mostafa Tavassolipour, Seyed Pooya ShariatpanahiFederated Learning Heterogeneous Federated Learning
School of Electrical and Computer Engineering College of Engineering, University of Tehran Tehran, Iran
FedDAP: Domain-Aware Prototype Learning for Federated Learning under Domain Shift
Apr 8, 2026Huy Q. Le, Loc X. Nguyen, Yu Qiao +3Federated Learning Distribution Shift
G-LAMP NEXUS Institute, Kyung Hee University · Kyung Hee University

Mar 15, 2026J/K move · Enter open · S save

Domain-Skewed Federated Learning with Feature Decoupling and Calibration

Huan Wang, Jun Shen, Jun Yan, Guansong Pang

School of Computing and Information Technology, University of Wollongong, Australia · School of Computing and Information Systems, Singapore Management University, Singapore

Federated Learning Heterogeneous Federated Learning Domain Generalization Representation Disentanglement

Open paper AskPDF arXiv

Federated learning (FL) allows distributed clients to collaboratively train a global model in a privacy-preserving manner. However, one major challenge is domain skew, where clients' data originating from diverse domains may hinder the aggregated global model from learning a consistent representation space, resulting in poor generalizable ability in multiple domains. In this paper, we argue that the domain skew is reflected in the domain-specific biased features of each client, causing the local model's representations to collapse into a narrow low-dimensional subspace. We then propose Federated Feature Decoupling and Calibration (

F^2

DC), which liberates valuable class-relevant information by calibrating the domain-specific biased features, enabling more consistent representations across domains. A novel component, Domain Feature Decoupler (DFD), is first introduced in

F^2

DC to determine the robustness of each feature unit, thereby separating the local features into domain-robust features and domain-related features. A Domain Feature Corrector (DFC) is further proposed to calibrate these domain-related features by explicitly linking discriminative signals, capturing additional class-relevant clues that complement the domain-robust features. Finally, a domain-aware aggregation of the local models is performed to promote consensus among clients. Empirical results on three popular multi-domain datasets demonstrate the effectiveness of the proposed

F^2

DC and the contributions of its two modules. Code is available at https://github.com/mala-lab/F2DC.

Abstract

Federated Learning

Explore similar work

Domain-Skewed Federated Learning with Feature Decoupling and Calibration

Joint Domain-Class Modeling for Federated Learning Under Feature Skew

FedDAP: Domain-Aware Prototype Learning for Federated Learning under Domain Shift