Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation

Lee En-Yi Hannah; Haozhi Cao; Yuecong Xu

cs.CVAug 29, 2026

Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation

Authors: Lee En-Yi Hannah, Haozhi Cao, Yuecong Xu

Organizations: National University of Singapore · The University of Tokyo

Abstract

Federated Video Domain Adaptation (FVDA) enables collaborative learning across distributed and non-IID video datasets while preserving privacy, but is under-explored due to challenges in aligning temporal information. We propose Multi-scalE Temporal domAin aLignment (METAL), a novel framework that leverages temporal information at multiple resolutions to improve cross-domain video action recognition with only model parameter transfers. METAL trains per-scale transformer encoders on source-clients, then performs independent knowledge voting at each temporal scale to generate robust pseudo-labels on the target-server. A novel $L_2$ variance penalty enforces cross-scale consistency during scale-based knowledge distillation, preventing a singular dominant scale. The late fusion aggregates features across different scales, where the fusion head is trained via knowledge distillation using confidence-weighted aggregation of scale-wise predictions, enabling the model to effectively exploit complementary temporal information for final predictions. Experiments on Epic-Kitchens-55 and Daily-DA demonstrate state-of-the-art performances, with gains up to 28.47% over current FDA methods. Ablation studies prove that multi-scale distillation and scale coordination are critical for effective temporal knowledge transfer.

Figures & tables

Figure 1: METAL Architecture. Source-clients train per-scale models independently before uploading to the target-server. For each temporal scale, a knowledge vote process aggregates source predictions into high-confidence pseudo-labels, forming a generalised teacher model. Target students minimize per-scale KL divergence with these pseudo-labels, while an $L2$ variance penalty enforces cross-scale consistency. The target fusion head is trained on concatenated multi-scale target features, distilled using confidence-weighted ensemble pseudo-labels as the teacher signal. After each communication round, federated averaging with consensus-based domain weights updates all models. Only 2 scales are shown for clarity.

Num. Segments ( $T$ )	Scale 1	Scale 4	Scale 16	Target Acc.
16	0.3101	0.3162	0.3029	0.3223
64	0.4240	0.4254	0.4230	0.4158
256 (Ours)	0.5216	0.4990	0.5082	0.5298
512	0.5226	0.5421	0.5164	0.5380

Table 1: Sensitivity analysis of the number of temporal segments (

T

) on Epic-Kitchens-55. Performance scales significantly with higher temporal resolution.

Source	Target	KD3A	FDAC	FedGP	Co-MDA	METAL
P08, P01	P22	0.4333	0.4466	0.4698	0.3921	0.5298
P22, P01	P08	0.3517	0.3425	0.4351	0.3597	0.4506
P22, P08	P01	0.4560	0.4667	0.4549	0.3963	0.4947
Avg. Target Acc.		0.4137	0.4186	0.4533	0.3827	0.4917
Avg. METAL Perf. Gain		0.1886	0.1746	0.0848	0.2847	–

Table 2: Comparison of METAL against current FDA methods on Epic-Kitchens-55.

Source	Target	KD3A	FDAC	Co-MDA	METAL
HMDB51, MIT	ARID	0.2490	0.1979	0.2284	0.2999
ARID, MIT	HMDB51	0.5930	0.5558	0.5814	0.5967
ARID, HMDB51	MIT	0.3250	0.3229	0.3912	0.3452
Avg. Target Acc.		0.3890	0.3589	0.4003	0.4139
Avg. METAL Perf. Gain		0.0641	0.1535	0.0339	–

Table 3: Comparison of METAL against current FDA methods on Daily-DA.

Table 4: Comprehensive ablation studies for METAL on Epic-Kitchens-55, averaged across all three tasks.

Parameter	Value	$S_{1}$	$S_{2}$	$S_{3}$	Target Acc.
Scale Spacing	[1, 2, 4]	0.5092	0.5051	0.5237	0.5257
	[1, 4, 16] (Ours)	0.5216	0.4990	0.5082	0.5298
	[1, 8, 64]	0.5246	0.5185	0.4004	0.5267
$L2$ Loss Weight	0.01	0.5257	0.5103	0.4867	0.5267
	0.05	0.5277	0.5072	0.4846	0.5267
	0.10 (Ours)	0.5216	0.4990	0.5082	0.5298

Table 5: Hyperparameter sensitivity study for METAL on Epic-Kitchens-55. Scale accuracies

S_{1},S_{2},S_{3}

correspond to the three temporal resolutions selected for each configuration.

Temporal Domain Adaptation

See all 10 papers

May 19, 2026

Return of Frustratingly Easy Unsupervised Video Domain Adaptation

Unsupervised video domain adaptation (UVDA) is a practical but under-explored problem. In this paper, we propose a frustratingly easy UVDA method, called MetaTrans. Specifically, MetaTrans adopts a concise learning objective that contains only two fundamental loss terms. Despite the simplicity of the learning objective, MetaTrans embodies an advanced UVDA idea, that is, handling the spatial and temporal divergence of cross-domain videos separately, through a subtle model architecture design. By implementing a temporal-static subtraction module, MetaTrans effectively removes spatial and temporal divergence. Extensive empirical evaluations, particularly on various cross-domain action recognition tasks, show substantial absolute adaptation performance enhancement and significantly superior relative performance gain compared with state-of-the-art UVDA baselines.

Pengfei Wei, Yiqun Sun, Zhiqiang Xu +2

1Magellan Technology Research Institute (MTRI), Japan · 2Mohamed bin Zayed University of Artificial Intelligence, UAE · 3Nanyang Technological University, Singapore.

Unsupervised Domain Adaptation Video Action Recognition

Ask PDF

Mar 23, 2026

FedCVU: Federated Learning for Cross-View Video Understanding

Federated learning (FL) has emerged as a promising paradigm for privacy-preserving multi-camera video understanding. However, applying FL to cross-view scenarios faces three major challenges: (i) heterogeneous viewpoints and backgrounds lead to highly non-IID client distributions and overfitting to view-specific patterns, (ii) local distribution biases cause misaligned representations that hinder consistent cross-view semantics, and (iii) large video architectures incur prohibitive communication overhead. To address these issues, we propose FedCVU, a federated framework with three components: VS-Norm, which preserves normalization parameters to handle view-specific statistics; CV-Align, a lightweight contrastive regularization module to improve cross-view representation alignment; and SLA, a selective layer aggregation strategy that reduces communication without sacrificing accuracy. Extensive experiments on action understanding and person re-identification tasks under a cross-view protocol demonstrate that FedCVU consistently boosts unseen-view accuracy while maintaining strong seen-view performance, outperforming state-of-the-art FL baselines and showing robustness to domain heterogeneity and communication constraints.

Shenghan Zhang, Run Ling, Ke Cao +2

Software College, Northeastern University, Shenyang, China · University of Science and Technology of China, Hefei, China · University of Chinese Academy of Sciences, Beijing, China +1

Federated Learning Video Understanding Non-IID Federated Learning Multi-View Consistency Federated Learning Aggregation

Ask PDF

Apr 10, 2026

Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation

Video Unsupervised Domain Adaptation (VUDA) poses a significant challenge in action recognition, requiring the adaptation of a model from a labeled source domain to an unlabeled target domain. Despite recent advances, existing VUDA methods often fall short of fully supervised performance, a key reason being the prevalence of static and uninformative backgrounds that exacerbate domain shifts. Additionally, prior approaches largely overlook computational efficiency, limiting real-world adoption. To address these issues, we propose Learnable Motion-Focused Tokenization (LMFT) for VUDA. LMFT tokenizes video frames into patch tokens and learns to discard low-motion, redundant tokens, primarily corresponding to background regions, while retaining motion-rich, action-relevant tokens for adaptation. Extensive experiments on three standard VUDA benchmarks across 21 domain adaptation settings show that our VUDA framework with LMFT achieves state-of-the-art performance while significantly reducing computational overhead. LMFT thus enables VUDA that is both effective and computationally efficient.

Tzu Ling Liu, Ian Stavness, Mrigank Rochan

University of Saskatchewan, Canada

Unsupervised Domain Adaptation Video Action Recognition Video Token Compression

Ask PDF

Abstract

Temporal Domain Adaptation

Explore similar work

Return of Frustratingly Easy Unsupervised Video Domain Adaptation

FedCVU: Federated Learning for Cross-View Video Understanding

Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation