One-shot federated unsupervised domain adaptation with Smoothed Knowledge Distillation and teacher refinement

Ali Abedi; Q. M. Jonathan Wu; Ning Zhang; Farhad Pourpanah

cs.CVOct 22, 2026

One-shot federated unsupervised domain adaptation with Smoothed Knowledge Distillation and teacher refinement

Authors: Ali Abedi, Q. M. Jonathan Wu, Ning Zhang, Farhad Pourpanah

Organizations: University of Windsor, Department of Electrical and Computer Engineering · Queen’s University, Department of Electrical and Computer Engineering

Abstract

Federated unsupervised domain adaptation (FUDA) addresses the challenge of adapting models to an unlabeled target domain using decentralized source domains while preserving data privacy. However, existing FUDA methods typically require multiple communication rounds, rely on complex aggregation strategies, and often struggle with noisy pseudo labels and inconsistent source knowledge. To address these challenges, we propose SKD-ETR, a novel one-shot FUDA framework that combines Smoothed Knowledge Distillation (SKD) and Exponential moving average-based Teacher Refinement (ETR). SKD trains a student model on the target domain using smoothed soft pseudo labels generated by the ensemble of source models. This reduces overconfidence, mitigates noise, and improves robustness. ETR further refines each source model by interpolating its parameters toward the student via exponential moving average (EMA), thereby transferring target-domain knowledge back to the teachers. This bidirectional refinement enhances pseudo-label quality and student generalization without additional communication overhead. SKD-ETR eliminates complex aggregation by initializing the student model randomly and performing a single-round distillation process. Extensive experiments on OfficeHome, Office-Caltech, and DomainNet demonstrate that SKD-ETR achieves competitive performance while being communication- and computation-efficient, and robust under noisy supervision. The code for this paper is available at https://github.com/A-Abedi/SKD_ETR.

Figures & tables

Model	FL rounds	Communication	$\sim$ Communication cost	Aggregation	Adaptation
FDAC [ 48 ]	R	Local models	$R\times M\times$ 85M Params	FedAvg	Domain alignment + semantic matching
FADA [ 29 ]	R	Frequent gradient sharing	$R\times M\times$ 45M Params	Source gradients aggregation	Adversarial feature disentanglement
FADE [ 19 ]	R	Local models + Global Model	$R\times M\times$ 45M Params	FedAvg	Adversarial debiasing
KD3A [ 13 ]	R	Local models + Global Model	$R\times M\times$ 23M Params	Consensus focus reweighting	Confidence gating + knowledge-vote ensemble
FDKD [ 45 ]	R	Local models + Global Model	$R\times M\times$ 23M Params	No aggregation step	Confidence gating + knowledge-vote ensemble
SEA+MSPL [ 2 ]	1	Bottleneck and head of local models	$1\times M\times$ 4M Params	Scaled-entropy weighting	Multi-source pseudo labeling + SSCE

Table 1: Comparison of representative one-shot and multi-round FUDA methods.

R

denotes the number of rounds and

M

the number of clients.

Figure 1: Overview of SKD-ETR. (a) Each client independently trains a source model on its private labeled dataset drawn from a distinct distribution. (b) The server initializes a randomly initialized student model for adaptation. (c) The source models act as teachers to generate smoothed pseudo labels for the unlabeled target-domain data, which are used to train a student model via KD with KL divergence. Then each source model is refined using an exponential moving average of the student model’s parameters to improve generalization without additional communication. 1 1 1 Distillation icons created by Prosymbols Premium - Flaticon

Figure 2: Illustration of the ETR process. Each source model initially generates pseudo-labels that reflect its own source-domain distribution, while the student, trained on target-domain pseudo-labels using SKD, represents a distribution closer to the true target domain. Through ETR, teachers are gradually pulled toward the student, which aligns their distributions more closely with the target and enables them to produce more target-aware pseudo-labels.

Figure 3: Sample images from different domains in different datasets. (a) DomainNet, (b) OfficeHome, and (c) Office-Caltech. Each row represents a category, and each column corresponds to a visual domain within that dataset.

Figure 4: t-SNE visualizations of the learned feature representations of the student model on the OfficeHome dataset before and after domain adaptation.

Domain	Flip Rate (↑)	KL Divergence (↑)
Art	0.9749	17.4327
Clipart	0.9835	15.8056
Product	0.9905	16.2890
Real World	0.9837	16.2571

Table 2: Quantitative analysis of decision boundary shifts before and after adaptation on the OfficeHome dataset using Flip Rate and KL Divergence.

	Model	OfficeHome					Office-Caltech
	Model	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ D	$\rightarrow$ W	Avg.
MSUDA	SImpAl [ 40 ]	73.4	62.4	81.0	82.7	74.8	95.6	94.6	100.0	100.0	97.5
	SSD [ 22 ]	72.5	64.5	81.2	83.2	75.4	95.8	95.0	100.0	99.1	97.5
	MDAMA [ 9 ]	77.7	69.9	85.6	87.1	80.0	96.9	96.7	100.0	100.0	98.4
MSFDA	SHOT [ 23 ]	82.9	82.8	59.3	72.2	74.3	96.4	96.2	98.5	99.7	97.7
	DECISION [ 3 ]	83.6	84.4	59.4	74.5	75.5	95.9	95.9	99.7	99.6	98.4

Table 3: Comparison of accuracy (%) on OfficeHome and Office-Caltech benchmarks across MSUDA, MSFDA, and FUDA methods.

\dagger

denotes one-shot FL.

	Model	$\rightarrow$ C	$\rightarrow$ I	$\rightarrow$ P	$\rightarrow$ Q	$\rightarrow$ R	$\rightarrow$ S	Avg.
MSUDA	SSD [ 22 ]	67.2	21.7	52.4	20.8	67.8	55.3	47.5
	SImpAl [ 40 ]	66.4	26.5	56.6	18.9	68.0	55.5	48.6
	MDAMA [ 9 ]	69.2	26.5	57.7	19.6	69.7	56.3	49.8
MSFDA	TransDA [ 47 ]	64.9	22.7	54.1	11.8	60.8	49.6	44.0
	SHOT [ 23 ]	61.7	22.2	52.6	12.2	67.7	48.6	44.2
	DECISION [ 3 ]	61.5	21.6	54.6	18.9	67.5	51.0	45.9

Table 4: Comparison of accuracy (%) on DomainNet benchmarks across MSUDA, MSFDA, and FUDA methods.

\dagger

denotes one-shot FL.

Model	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.
Ours	84.1	76.2	91.3	92.1	85.9
Ours w/o ETR	83.3	73.0	91.2	91.9	84.7
Ours w/o SKD	81.7	72.4	90.1	90.8	83.7
Ours w/o ETR & SKD, w/ KL Divergence	80.9	71.6	89.7	90.0	83.0

Table 5: Ablation results showing the impact of removing SKD and ETR on target domain performance.

Initialization Strategy	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.
Random initialization (Ours)	84.1	76.2	91.3	92.1	85.9
Averaging	82.8	74.0	90.7	91.7	84.8
Lowest-entropy source model	83.4	73.4	88.1	90.8	83.9
Random source model	83.6	71.9	88.6	90.7	83.7

Table 6: Ablation study of our method with different student model initialization strategies.

Pseudo Labeling Strategy	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.
SKD (Ours)	84.1	76.2	91.3	92.1	85.9
Entropy Weighting [ 24 ]	83.8	73.9	89.7	89.1	84.1
Confidence Weighting [ 53 ]	83.8	72.6	87.5	90.3	83.6
Confidence Thresholding [ 31 ]	76.4	62.6	85.1	85.8	77.5

Table 7: Performance comparison of our model using different pseudo labeling strategies.

Backbone	Method	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.
Resnet-50 [ 17 ]	w/o ETR & SKD, w/ KL Divergence	64.6	45.8	74.2	78.2	65.7
Resnet-50 [ 17 ]	SKD-ETR	66.0	47.2	77.4	80.3	67.7
Resnet-101 [ 17 ]	w/o ETR & SKD, w/ KL Divergence	67.4	51.9	77.8	80.6	69.4
Resnet-101 [ 17 ]	SKD-ETR	71.4	52.6	78.9	80.9	71.0
ViT [ 11 ]	w/o ETR & SKD, w/ KL Divergence	66.9	56.2	78.6	80.5	70.5
ViT [ 11 ]	SKD-ETR	69.9	59.3	82.2	82.8	73.5

Table 8: Performance comparison of our model across different backbones.

Figure 5: Sensitivity analysis of the smoothing factor

\epsilon

in SKD.

$\epsilon$ Strategy	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.
Fixed (Ours)	84.1	76.2	91.3	92.1	85.9
Entropy-adaptive	83.3	76.5	91.0	91.9	85.6
Time-adaptive	83.4	71.8	91.2	91.3	84.4

Table 9: Performance comparison on different label smoothing strategies for pseudo-label refinement.

Figure 6: Sensitivity analysis of our model across different values of

\alpha

Gaussian noise $\sigma$	Method	$\rightarrow$ A	$\rightarrow$ C	$\rightarrow$ P	$\rightarrow$ R	Avg.
0 (No Noise)	FedAvg	66.4	63.0	82.7	83.8	74.0
0 (No Noise)	SKD-ETR	84.1	76.2	91.3	92.1	85.9
0.01	FedAvg	63.8	61.5	82.5	81.6	72.4 (-1.6)
0.01	SKD-ETR	83.8	76.1	90.8	91.5	85.6 (-0.3)
0.02	FedAvg	49.0	53.0	63.7	81.6	61.8 (-12.2)
0.02	SKD-ETR	83.2	75.4	90.7	91.3	85.2 (-0.7)

Table 10: Sensitivity analysis of SKD-ETR to Gaussian noise injected into source model parameters before transmission to the server.

Figure 7: Target-domain accuracy of each source model before and after adaptation process.

Target Domain	Before Adaptation	After Adaptation
Art	2.15	0.00044
Clipart	2.55	0.00044
Product	1.45	0.00048
Real World	1.67	0.00049

Table 11: Average pairwise KL divergence between teacher model predictions on the target domain before and after adaptation.

Model	Training Time (s)	Peak Mem (GB)	Params (MB)	FLOPs (GF)	CO 2 Emission (kg)
Art	1559.0	0.81	363.7	70.7	0.015
Clipart	1559.0	0.81	363.7	70.7	0.023
Product	1569.5	0.81	363.7	70.7	0.023
Real World	1554.8	0.81	363.7	70.7	0.023
Average (Teacher)	1560.6	0.81	363.7	70.7	0.021
Student (Target)	2094.2	1.25	363.7	70.7	0.051

Table 12: Computational overhead and resource usage for SKD-ETR across teacher and student models on the OfficeHome dataset.

Figure 8: Fréchet distances between DomainNet domains.

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 183 papers

FedDAF: Federated Domain Adaptation Using Model Functional Distance
Sep 15, 2025Mrinmay Sen, Sidhant Nair, C Krishna MohanFederated Learning Aggregation Domain Adaptation
Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, Hyderabad, 502285, Telangana, India. · Department of Mechanical Engineering, Indian Institute of Technology Delhi, New Delhi, 110016, Delhi, India.
A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning
Jul 3, 2026Sushant Dagaji Desale, Rahul Mishra, Ashutosh Kumar SinhaAdversarial Training Unsupervised Domain Adaptation
Indian Institute of Technology Patna
DeFed-GMM-DaDiL: A Decentralized Federated Framework for Domain Adaptation
May 5, 2026Rebecca Clain, Eduardo Fernandes Montesuma, Fred Ngole MboulaMulti-Source Domain Adaptation Domain Adaptation
Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Sigma Nova

Sep 15, 2025J/K move · Enter open · S save

FedDAF: Federated Domain Adaptation Using Model Functional Distance

Mrinmay Sen, Sidhant Nair, C Krishna Mohan

Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, Hyderabad, 502285, Telangana, India. · Department of Mechanical Engineering, Indian Institute of Technology Delhi, New Delhi, 110016, Delhi, India.

Federated Learning Aggregation Domain Adaptation Model Merging Federated Learning

Open paper AskPDF arXiv

Federated Domain Adaptation (FDA) improves model performance at a target client by collaborating with source clients while preserving data privacy. FDA faces two key challenges: domain shift between source and target data, and limited labeled data at the target, a common constraint when a new site joins a federation before it has accumulated its own labeled data, as in clinical deployments. Most existing methods address domain shift alone, assuming ample target data; those that also tackle data scarcity still fail to prioritize source information according to the target's specific objective. We propose FedDAF, which addresses both challenges through similarity-based aggregation of the global source and target models, using their model functional distance, computed from the angle between their mean gradient fields on target data and normalized via a Gompertz function. The global source model itself is formed using a distance-based weighted average, giving greater weight to source models closer to the target model. Experiments on real-world datasets show FedDAF outperforms existing federated learning (FL), personalized FL, and FDA methods in test accuracy.

Abstract

Unsupervised Domain Adaptation

Explore similar work

FedDAF: Federated Domain Adaptation Using Model Functional Distance

A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning

DeFed-GMM-DaDiL: A Decentralized Federated Framework for Domain Adaptation