Domain-Skewed Federated Learning with Feature Decoupling and Calibration

Huan Wang; Jun Shen; Jun Yan; Guansong Pang

cs.LGMar 15, 2026

Domain-Skewed Federated Learning with Feature Decoupling and Calibration

Authors: Huan Wang, Jun Shen, Jun Yan, Guansong Pang

Organizations: School of Computing and Information Technology, University of Wollongong, Australia · School of Computing and Information Systems, Singapore Management University, Singapore

Abstract

Federated learning (FL) allows distributed clients to collaboratively train a global model in a privacy-preserving manner. However, one major challenge is domain skew, where clients' data originating from diverse domains may hinder the aggregated global model from learning a consistent representation space, resulting in poor generalizable ability in multiple domains. In this paper, we argue that the domain skew is reflected in the domain-specific biased features of each client, causing the local model's representations to collapse into a narrow low-dimensional subspace. We then propose Federated Feature Decoupling and Calibration ( $F^2$ DC), which liberates valuable class-relevant information by calibrating the domain-specific biased features, enabling more consistent representations across domains. A novel component, Domain Feature Decoupler (DFD), is first introduced in $F^2$ DC to determine the robustness of each feature unit, thereby separating the local features into domain-robust features and domain-related features. A Domain Feature Corrector (DFC) is further proposed to calibrate these domain-related features by explicitly linking discriminative signals, capturing additional class-relevant clues that complement the domain-robust features. Finally, a domain-aware aggregation of the local models is performed to promote consensus among clients. Empirical results on three popular multi-domain datasets demonstrate the effectiveness of the proposed $F^2$ DC and the contributions of its two modules. Code is available at https://github.com/mala-lab/F2DC.

Figures & tables

Figure 1: Visualization of the singular values of the feature covariance matrix on the PACS [ 19 ] dataset from Vanilla FL [ 31 ] ( Top ) and our method ( Bottom ). Under domain-skewed FL, the Vanilla method suffers from severe dimensional collapse, i.e . , representations are largely biased and reside in a domain-specific lower-dimensional subspace. Our method effectively mitigates this collapse, as reflected in more uniformly distributed singular values and significantly fewer values tending to zero.

Figure 2: Grad-CAM [ 35 ] visualization of the features obtained by three methods—Vanilla method [ 31 ] ( Top ), domain-skewed method FDSE [ 48 ] ( Middle ), and $F^{2}$ DC ( Bottom )—for samples from different domains on a randomly selected $\text{Client}_{Photo}$ belonging to the photo domain. For our method $F^{2}$ DC, we further provide a qualitative example about how its two modules work: local features are first decoupled via DFD into domain-robust features and domain-related features. Then, the domain-related features (which entangle valuable class-relevant information with domain biases) are calibrated via DFC, resulting in the Correction features. By incorporating the domain-robust and the corrected features, $F^{2}$ DC captures more holistic class-relevant clues, enabling the local model to yield more consistent decisions for samples with the same semantics across different domains.

Figure 3: Overview of the proposed $F^{2}$ DC approach. During client-side local training, we initially separate local features into domain-robust features and domain-related features via a Domain Feature Decoupler (DFD) (Sec. 4.1 ), and then rectify the domain-related features via a Domain Feature Corrector (DFC) (Sec. 4.2 ). During server-side global aggregation, we perform domain-aware aggregation (Sec. 4.3 ) of the global model by incorporating each client’s domain discrepancy $\mathbf{p}_{k}$ to promote more consensus among local clients.

	Office-Caltech						PACS
Methods	Caltech	Amazon	Webcam	DSLR	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$	Photo	Art-Paint	Cartoon	Sketch	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$
FedAvg [AISTATS’17]	59.82	65.26	51.72	46.67	55.86	8.27	60.98	52.49	74.63	77.46	66.39	11.74
FedProx [MLSys’20]	60.18	66.11	54.96	47.30	57.13	7.98	61.88	57.40	76.34	80.14	68.94	11.00
MOON [CVPR’21]	57.14	63.16	44.48	40.87	51.41	10.49	59.72	50.29	70.35	70.22	62.64	9.63
FPL [CVPR’23]	60.05	65.53	52.90	64.45	60.73	5.75	66.67	58.38	79.75	77.59	70.59	9.95
FedTGP [AAAI’24]	61.28	63.84	53.37	63.68	60.54	4.92	60.68	56.06	75.27	78.03	67.51	10.78

Table 1: Comparison results on Office-Caltech and PACS. Best results are in red bold , with second best in blue bold .

	Digits
Methods	MNIST	USPS	SVHN	SYN	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$
FedAvg	96.04	89.84	88.04	51.05	81.24	20.42
FedProx	96.13	89.04	87.56	51.00	80.93	20.30
MOON	94.70	89.64	87.68	34.40	76.60	28.29
FPL	96.12	89.94	89.15	49.15	81.09	21.52
FedTGP	95.38	87.49	87.37	49.40	79.91	20.68

Table 2: Comparison results on Digits, continuing Table 1 .

Methods	P	AP	Ct	Sk	AVG $\boldsymbol{\uparrow}$
FedAvg	74.86 ↑13.88	65.90 ↑13.41	79.70 ↑5.07	80.86 ↑3.40	75.33 ↑8.94
FPL	76.85 ↑10.18	64.46 ↑6.08	80.02 ↑0.27	80.75 ↑3.16	75.52 ↑4.93
FedHEAL	75.56 ↑5.17	65.95 ↑0.13	81.27 ↑0.70	77.48 ↑0.87	75.06 ↑1.72
FDSE	72.25 ↑2.98	65.99 ↑0.53	80.56 ↑1.91	80.36 ↑1.24	74.79 ↑1.66

Table 3: Modularity of

F^{2}

DC on PACS, where we plug-in our proposed DFD and DFC into four strong baselines.

Figure 4: T-SNE visualization on PACS. Each color means one class, each shape means one domain, stars are semantic centers.

Methods	$\text{Upload}_{\times 10^{7}}$	Comm.	$\text{FLOPs}_{\times 10^{10}}$	$\text{Time}_{train}$
FedAvg	5.43	193.72MB	16.31	176.94s
FPL	5.45	197.30MB	19.45	215.61s
FedHEAL	5.43	193.72MB	16.70	186.92s
$F^{2}$ DC	5.43	193.72MB	16.46	180.67s

Table 4: Efficiency comparison per round (

10

clients) on PACS.

Figure 5: Comparison of convergence in average accuracy on Office-Caltech ( Left ) and PACS ( Right ). Zoom in for details.

Figure 6: Hyper-parameter study with different

\tau

of Eq. ( 5 ).

$\sigma$	0.01	0.05	0.1	0.3	0.5	1.0	5.0
AVG $\boldsymbol{\uparrow}$	74.81	75.19	76.47	75.34	74.40	74.15	73.86
STD $\boldsymbol{\downarrow}$	6.48	6.32	5.83	6.35	6.71	7.05	7.23

Table 5: Hyper-parameter study with

\sigma

of Eq. ( 3 ) on PACS.

Figure 7: Hyper-parameter study with

\alpha

and

\beta

of Eq. ( 11 ).

Figure 8: Hyper-parameter study with

\lambda_{1}

and

\lambda_{2}

of Eq. ( 9 ).

$\mathcal{L}_{DFD}$	$\mathcal{L}_{DFC}$	DaA	C	A	W	D	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$
			59.82	65.26	51.72	46.67	55.86	8.27
		✓	60.25	65.79	54.39	50.33	57.69	6.76
✓		✓	66.68	67.53	56.55	60.02	62.70	5.30
	✓	✓	63.71	68.21	59.07	67.62	64.65	4.22
✓	✓		60.84	67.79	62.10	70.45	65.29	4.57
✓	✓	✓	62.95	68.42	64.79	71.12	66.82	3.65

Table 6: Ablation study of key modules in our

F^{2}

DC on Office-Caltech ( Top ) and PACS ( Bottom ). Please see details in Sec. 5.3 .

Protocols	P	AP	Ct	Sk	AVG $\boldsymbol{\uparrow}$	STD $\boldsymbol{\downarrow}$
$f^{\scalebox{0.66}{$ \bm{+} $}}$	74.21	67.52	78.35	80.47	75.13	5.90
$f^{\scalebox{0.66}{$ \bm{-} $}}$	59.45	50.09	52.56	69.37	57.87	8.63
$f^{\scalebox{0.88}{$ \star $}}$	74.83	64.05	75.80	79.29	73.49	6.71
$\widetilde{f}$	75.15	68.71	79.91	82.11	76.47	5.83

Table 7: Ablation study of

F^{2}

DC’s various features on PACS.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 921 papers

Joint Domain-Class Modeling for Federated Learning Under Feature Skew
Sep 19, 2026Sina Najafi, Mostafa Tavassolipour, Seyed Pooya ShariatpanahiFederated Learning Heterogeneous Federated Learning
School of Electrical and Computer Engineering College of Engineering, University of Tehran Tehran, Iran
FedDAP: Domain-Aware Prototype Learning for Federated Learning under Domain Shift
Apr 8, 2026Huy Q. Le, Loc X. Nguyen, Yu Qiao +3Federated Learning Distribution Shift
G-LAMP NEXUS Institute, Kyung Hee University · Kyung Hee University
FedDAF: Federated Domain Adaptation Using Model Functional Distance
Sep 15, 2025Mrinmay Sen, Sidhant Nair, C Krishna MohanFederated Learning Aggregation Domain Adaptation
Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, Hyderabad, 502285, Telangana, India. · Department of Mechanical Engineering, Indian Institute of Technology Delhi, New Delhi, 110016, Delhi, India.

Sep 19, 2026J/K move · Enter open · S save

Joint Domain-Class Modeling for Federated Learning Under Feature Skew

Sina Najafi, Mostafa Tavassolipour, Seyed Pooya Shariatpanahi

School of Electrical and Computer Engineering College of Engineering, University of Tehran Tehran, Iran

Federated Learning Heterogeneous Federated Learning Clustered Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) enables collaborative model training without centralizing private data, but performance often degrades under feature skew: clients share labels while the conditional input distributions

p_i(x\!\mid\!y)

vary due to latent, client-specific appearance factors. We propose Joint Domain-Class Federated Learning (JDFL), a lightweight, optimizer-agnostic extension that makes this latent domain variation usable without sharing raw data. JDFL first infers domain clusters called pseudo-domains from brief local update signals. It then expands the classifier head to output

M\times C

, joint (domain-class) logits. This allows the model to represent domain-conditioned appearance while keeping a shared backbone. To train the expanded head we introduce two complementary supervision strategies based on simple intuitions: a similarity-aware soft-labeling that transfers evidence between nearby inferred domains while allowing domain-specific specialization, and a per-sample randomized target assignment that perturbs supervision across the joint outputs and serves as a low-cost training-time regularizer. JDFL integrates with existing standard FL methods (e.g., FedAvg, SCAFFOLD) with minimal changes. Empirically, both supervision modes consistently improve global test accuracy on standard domain-shifted image benchmarks; ablations and sensitivity studies show the gains stem from the proposed supervision and parametrization rather than mere capacity increase.

Abstract

Federated Learning

Explore similar work

Joint Domain-Class Modeling for Federated Learning Under Feature Skew

FedDAP: Domain-Aware Prototype Learning for Federated Learning under Domain Shift

FedDAF: Federated Domain Adaptation Using Model Functional Distance