Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation

Tzu Ling Liu; Ian Stavness; Mrigank Rochan

cs.CVApr 10, 2026

Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation

Authors: Tzu Ling Liu, Ian Stavness, Mrigank Rochan

Organizations: University of Saskatchewan, Canada

Abstract

Video Unsupervised Domain Adaptation (VUDA) poses a significant challenge in action recognition, requiring the adaptation of a model from a labeled source domain to an unlabeled target domain. Despite recent advances, existing VUDA methods often fall short of fully supervised performance, a key reason being the prevalence of static and uninformative backgrounds that exacerbate domain shifts. Additionally, prior approaches largely overlook computational efficiency, limiting real-world adoption. To address these issues, we propose Learnable Motion-Focused Tokenization (LMFT) for VUDA. LMFT tokenizes video frames into patch tokens and learns to discard low-motion, redundant tokens, primarily corresponding to background regions, while retaining motion-rich, action-relevant tokens for adaptation. Extensive experiments on three standard VUDA benchmarks across 21 domain adaptation settings show that our VUDA framework with LMFT achieves state-of-the-art performance while significantly reducing computational overhead. LMFT thus enables VUDA that is both effective and computationally efficient.

Figures & tables

Figure 1: Overview of LMFT. For both source and target videos, LMFT tokenizes frames into patch tokens, computes the L1 distance between consecutive temporal tokens, and discards those with differences below a learnable threshold $\tau$ , which primarily correspond to redundant and static background regions. The remaining motion-rich action-relevant tokens are then fed into the ViT-based DA framework.

	Method	Accuracy (Top-1%)
	Method	H→A	M→A	K→A	A→H	M→H	K→H	H→M	A→M	K→M	M→K	H→K	A→K	Avg.
	Source Only	45.7	55.2	38.1	51.7	68.8	47.1	47.5	46.5	37.0	79.4	57.0	53.9	52.3
ZS	CLIP (ViT-B/16) [ 21 ]	36.5	36.5	36.5	60.0	60.0	60.0	48.5	48.5	48.5	68.1	68.1	68.1	53.3
SFVUDA	ATCoN [ 31 ]	17.9	27.2	17.2	26.7	47.3	48.2	30.7	17.2	32.5	57.7	48.5	31.0	33.5
	EXTERN [ 33 ]	26.2	18.1	23.9	26.2	53.7	55.8	40.7	18.2	35.2	68.1	57.6	51.4	39.6
	STHC [ 16 ]	15.5	48.7	34.8	18.4	56.3	76.6	13.8	39.8	50.1	44.6	27.3	44.7	39.2

Table 1: VUDA results on Daily-DA . Colored rows show our results, and the results in other rows are taken from [ 16 ] and [ 22 ] .

	Method	Accuracy (Top-1%)
	Method	H→U	U→H	Avg.
	Source Only	93.3	81.9	87.6
ZS	CLIP (ViT-B/16) [ 21 ]	88.8	91.7	90.3
SFVUDA	ATCoN [ 31 ]	85.3	79.7	82.5
	EXTERN [ 33 ]	91.9	88.9	90.4
	STHC [ 16 ]	92.1	90.9	91.5

Table 2: VUDA results on UCF

\leftrightarrow

HMDB full . Rows in colors are our experimental results, and others are from [ 16 ] and [ 22 ] .

Method	Accuracy (Top-1%)
Method	KT→C1	KT→C2	KT→C3	KT→C4	KT→C5	KT→C6	KT→C7	Avg.
Source Only	5.6	8.7	100.0	65.9	89.5	100.0	100.0	67.1
MM-SADA [ 20 ]	11.0	0.0	100.0	72.9	68.4	100.0	93.2	62.6
Zhang et al. [ 35 ]	16.6	8.5	100.0	75.5	94.6	100.0	96.5	67.3
UNITE [ 22 ]	5.6	8.7	100.0	100.0	94.7	100.0	100.0	76.4
Ours w/o LMFT	77.8	69.6	100.0	75.6	79.0	100.0	100.0	84.8

Table 3: VUDA results on ActorShift . Colored rows indicate our experimental results. Uncolored baseline results are from [ 35 ] , while UNITE [ 22 ] accuracy is from our run of their public code.

	Model	Accuracy ↑	Train time ↓	Speedup ↑
M→H	ViT-B/16 [ 9 ]	72.1	3810s	1.0 $\times$
	Random	71.7	3209s	1.2 $\times$
	ToMe [ 2 ]	72.5	4117s	0.9 $\times$
	PruMerge [ 24 ]	71.7	3630s	1.1 $\times$
	DivPrune [ 1 ]	72.9	16961s	0.22 $\times$
	LMFT (Ours)	74.2	2784s	1.4 $\times$

Table 4: Training efficiency of token reduction methods.

	Model	Accuracy ↑	Clips/s ↑	GFLOPs ↓	Cost ↓
M→H	ViT-B/16 [ 9 ]	72.1	3.8	266	1.00 $\times$
	Random	71.7	3.6	216	0.81 $\times$
	ToMe [ 2 ]	72.5	1.8	242	0.91 $\times$
	PruMerge [ 24 ]	71.7	3.5	493	1.85 $\times$
	DivPrune [ 1 ]	72.9	0.2	223	0.84 $\times$
	LMFT (Ours)	74.2	3.9	217	0.82 $\times$

Table 5: Inference efficiency of token reduction methods.

	Model	Accuracy ↑	Train time ↓	GFLOPs ↓
M→H	ViT-B/16 [ 9 ]	72.1	3810s	266
	DALL-V [ 34 ]	58.3	15526s	291
	UNITE [ 22 ]	71.7	27426s	358
	Ours	74.2	2784s	217
H→M	ViT-B/16 [ 9 ]	57.5	1211s	266
	DALL-V [ 34 ]	46.8	18368s	291

Table 6: Efficiency comparison of VUDA methods.

Figure 2: Visualization of LMFT on four videos (two left, two right). Each video has three rows: original frames, motion differences, and LMFT processed frames. LMFT selects action-relevant patches, dropping static or low-motion background patches for effective DA.

Confidence Threshold $\gamma_{c}$	Accuracy (%)
Confidence Threshold $\gamma_{c}$	M $\rightarrow$ H	H $\rightarrow$ M
0.0	68.8	54.3
0.2	67.5	57.0
0.4	68.8	52.7
0.6	72.5	57.7
0.8	74.2	60.0

Table 7: Impact of threshold

\gamma_{c}

in confidence-based filtering.

	Model	$N_{t}=2$	$N_{t}=4$	$N_{t}=6$	$N_{t}=8$
M→H	ViT-B/16 [ 9 ]	64.2	69.2	70.4	72.1
	Random	63.3	67.5	70.4	71.7
	UNITE [ 22 ]	63.3	70.4	68.8	71.7
	Ours	65.4	71.3	73.3	74.2
H→M	ViT-B/16 [ 9 ]	48.0	50.0	53.5	57.5
	Random	46.7	52.0	52.5	56.5

Table 8: Impact of different temporal resolutions on VUDA.

Reward hyperparameter $\lambda_{\mathcal{L}}$	Accuracy (%)
Reward hyperparameter $\lambda_{\mathcal{L}}$	M $\rightarrow$ H	H $\rightarrow$ M
0.1	73.3	56.7
1.0	73.8	57.0
10.0	74.2	60.0
100.0	72.8	58.5

Table 9: Effect of varying

\lambda_{\mathcal{L}}

in Eq. 7 .

M→H	Gumbel-Softmax	1.31	34.58	3275	73.3
	Method	Memory (GiB) ↓		Time ↓	Accuracy ↑
	Method	Min	Max	Time ↓	Accuracy ↑
M→H	LMFT(Ours)	1.31	30.03	2784	74.2
H→M	Gumbel-Softmax	1.31	34.58	1130	58.8
H→M	LMFT(Ours)	1.31	28.96	890	60.0

Table 10: Gumbel-Softmax vs. RL in LMFT.

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 179 papers

May 19, 2026

Return of Frustratingly Easy Unsupervised Video Domain Adaptation

Unsupervised video domain adaptation (UVDA) is a practical but under-explored problem. In this paper, we propose a frustratingly easy UVDA method, called MetaTrans. Specifically, MetaTrans adopts a concise learning objective that contains only two fundamental loss terms. Despite the simplicity of the learning objective, MetaTrans embodies an advanced UVDA idea, that is, handling the spatial and temporal divergence of cross-domain videos separately, through a subtle model architecture design. By implementing a temporal-static subtraction module, MetaTrans effectively removes spatial and temporal divergence. Extensive empirical evaluations, particularly on various cross-domain action recognition tasks, show substantial absolute adaptation performance enhancement and significantly superior relative performance gain compared with state-of-the-art UVDA baselines.

Pengfei Wei, Yiqun Sun, Zhiqiang Xu +2

1Magellan Technology Research Institute (MTRI), Japan · 2Mohamed bin Zayed University of Artificial Intelligence, UAE · 3Nanyang Technological University, Singapore.

Unsupervised Domain Adaptation Video Action Recognition

Ask PDF

Aug 29, 2026

Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation

Federated Video Domain Adaptation (FVDA) enables collaborative learning across distributed and non-IID video datasets while preserving privacy, but is under-explored due to challenges in aligning temporal information. We propose Multi-scalE Temporal domAin aLignment (METAL), a novel framework that leverages temporal information at multiple resolutions to improve cross-domain video action recognition with only model parameter transfers. METAL trains per-scale transformer encoders on source-clients, then performs independent knowledge voting at each temporal scale to generate robust pseudo-labels on the target-server. A novel

L_2

variance penalty enforces cross-scale consistency during scale-based knowledge distillation, preventing a singular dominant scale. The late fusion aggregates features across different scales, where the fusion head is trained via knowledge distillation using confidence-weighted aggregation of scale-wise predictions, enabling the model to effectively exploit complementary temporal information for final predictions. Experiments on Epic-Kitchens-55 and Daily-DA demonstrate state-of-the-art performances, with gains up to 28.47% over current FDA methods. Ablation studies prove that multi-scale distillation and scale coordination are critical for effective temporal knowledge transfer.

Lee En-Yi Hannah, Haozhi Cao, Yuecong Xu

National University of Singapore · The University of Tokyo

Temporal Domain Adaptation Unsupervised Domain Adaptation Video Action Recognition Federated Learning Federated Knowledge Distillation Multi-Scale Feature Fusion

Ask PDF

Jul 27, 2026

Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts

Deep learning models have achieved state-of-the-art performance in several computer vision tasks. However, they experience severe performance degradation when applied to real-world scenarios due to unanticipated distribution shifts. Test-Time Adaptation (TTA) attempts to solve this problem by using unlabeled data from the target domain to dynamically adapt to the test distribution at inference time, without access to the source data. However, TTA remains a challenging problem when adapting to continuous, temporally correlated data, such as videos, and in scenarios where the target domain contains severe domain shifts. For this reason, few works in the literature explore TTA for videos under such extreme conditions. To overcome these limitations, we propose Test-time Adaptation via Dual Distillation (TADD), an online adaptation framework that relies on a lightweight projection adapter to bridge the domain gap. The adapter module is pre-trained on the source domain and then adapted to the target using our proposed complementary losses: (i) zero-shot distillation, which encourages alignment with the domain-agnostic features from a pre-trained vision-language model (VLM); and (ii) target distillation, which retains the source domain discriminative knowledge encoded in the pre-trained adapter. Built upon a frozen CLIP backbone, our method introduces this lightweight projection adapter as the sole updatable component during inference. We conducted extensive evaluations on three well-known video action recognition benchmarks: UCF-HMDB, Daily-DA, and Sports-DA. Our experiments in the closed-set scenario demonstrate that our method consistently outperforms state-of-the-art TTA baselines. Notably, our TTA approach improves upon previous methods by up to +3.81% on UCF-HMDB, +2.63% on Daily-DA, and +3.03% on Sports-DA.

André Sacilotti, Samuel Felipe dos Santos, Jurandy Almeida

University of São Paulo São Carlos, São Paulo, Brazil · Federal University of São Carlos Sorocaba, São Paulo, Brazil

Source-Free Domain Adaptation Test-Time Adaptation Video Action Recognition

Ask PDF

Abstract

Unsupervised Domain Adaptation

Explore similar work

Return of Frustratingly Easy Unsupervised Video Domain Adaptation

Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation

Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts