Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation

Tzu Ling Liu; Ian Stavness; Mrigank Rochan

cs.CVApr 10, 2026

Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation

Authors: Tzu Ling Liu, Ian Stavness, Mrigank Rochan

Organizations: University of Saskatchewan, Canada

Abstract

Video Unsupervised Domain Adaptation (VUDA) poses a significant challenge in action recognition, requiring the adaptation of a model from a labeled source domain to an unlabeled target domain. Despite recent advances, existing VUDA methods often fall short of fully supervised performance, a key reason being the prevalence of static and uninformative backgrounds that exacerbate domain shifts. Additionally, prior approaches largely overlook computational efficiency, limiting real-world adoption. To address these issues, we propose Learnable Motion-Focused Tokenization (LMFT) for VUDA. LMFT tokenizes video frames into patch tokens and learns to discard low-motion, redundant tokens, primarily corresponding to background regions, while retaining motion-rich, action-relevant tokens for adaptation. Extensive experiments on three standard VUDA benchmarks across 21 domain adaptation settings show that our VUDA framework with LMFT achieves state-of-the-art performance while significantly reducing computational overhead. LMFT thus enables VUDA that is both effective and computationally efficient.

Figures & tables

Figure 1: Overview of LMFT. For both source and target videos, LMFT tokenizes frames into patch tokens, computes the L1 distance between consecutive temporal tokens, and discards those with differences below a learnable threshold $\tau$ , which primarily correspond to redundant and static background regions. The remaining motion-rich action-relevant tokens are then fed into the ViT-based DA framework.

	Method	Accuracy (Top-1%)
	Method	H→A	M→A	K→A	A→H	M→H	K→H	H→M	A→M	K→M	M→K	H→K	A→K	Avg.
	Source Only	45.7	55.2	38.1	51.7	68.8	47.1	47.5	46.5	37.0	79.4	57.0	53.9	52.3
ZS	CLIP (ViT-B/16) [ 21 ]	36.5	36.5	36.5	60.0	60.0	60.0	48.5	48.5	48.5	68.1	68.1	68.1	53.3
SFVUDA	ATCoN [ 31 ]	17.9	27.2	17.2	26.7	47.3	48.2	30.7	17.2	32.5	57.7	48.5	31.0	33.5
	EXTERN [ 33 ]	26.2	18.1	23.9	26.2	53.7	55.8	40.7	18.2	35.2	68.1	57.6	51.4	39.6
	STHC [ 16 ]	15.5	48.7	34.8	18.4	56.3	76.6	13.8	39.8	50.1	44.6	27.3	44.7	39.2

Table 1: VUDA results on Daily-DA . Colored rows show our results, and the results in other rows are taken from [ 16 ] and [ 22 ] .

	Method	Accuracy (Top-1%)
	Method	H→U	U→H	Avg.
	Source Only	93.3	81.9	87.6
ZS	CLIP (ViT-B/16) [ 21 ]	88.8	91.7	90.3
SFVUDA	ATCoN [ 31 ]	85.3	79.7	82.5
	EXTERN [ 33 ]	91.9	88.9	90.4
	STHC [ 16 ]	92.1	90.9	91.5

Table 2: VUDA results on UCF

\leftrightarrow

HMDB full . Rows in colors are our experimental results, and others are from [ 16 ] and [ 22 ] .

Method	Accuracy (Top-1%)
Method	KT→C1	KT→C2	KT→C3	KT→C4	KT→C5	KT→C6	KT→C7	Avg.
Source Only	5.6	8.7	100.0	65.9	89.5	100.0	100.0	67.1
MM-SADA [ 20 ]	11.0	0.0	100.0	72.9	68.4	100.0	93.2	62.6
Zhang et al. [ 35 ]	16.6	8.5	100.0	75.5	94.6	100.0	96.5	67.3
UNITE [ 22 ]	5.6	8.7	100.0	100.0	94.7	100.0	100.0	76.4
Ours w/o LMFT	77.8	69.6	100.0	75.6	79.0	100.0	100.0	84.8

Table 3: VUDA results on ActorShift . Colored rows indicate our experimental results. Uncolored baseline results are from [ 35 ] , while UNITE [ 22 ] accuracy is from our run of their public code.

	Model	Accuracy ↑	Train time ↓	Speedup ↑
M→H	ViT-B/16 [ 9 ]	72.1	3810s	1.0 $\times$
	Random	71.7	3209s	1.2 $\times$
	ToMe [ 2 ]	72.5	4117s	0.9 $\times$
	PruMerge [ 24 ]	71.7	3630s	1.1 $\times$
	DivPrune [ 1 ]	72.9	16961s	0.22 $\times$
	LMFT (Ours)	74.2	2784s	1.4 $\times$

Table 4: Training efficiency of token reduction methods.

	Model	Accuracy ↑	Clips/s ↑	GFLOPs ↓	Cost ↓
M→H	ViT-B/16 [ 9 ]	72.1	3.8	266	1.00 $\times$
	Random	71.7	3.6	216	0.81 $\times$
	ToMe [ 2 ]	72.5	1.8	242	0.91 $\times$
	PruMerge [ 24 ]	71.7	3.5	493	1.85 $\times$
	DivPrune [ 1 ]	72.9	0.2	223	0.84 $\times$
	LMFT (Ours)	74.2	3.9	217	0.82 $\times$

Table 5: Inference efficiency of token reduction methods.

	Model	Accuracy ↑	Train time ↓	GFLOPs ↓
M→H	ViT-B/16 [ 9 ]	72.1	3810s	266
	DALL-V [ 34 ]	58.3	15526s	291
	UNITE [ 22 ]	71.7	27426s	358
	Ours	74.2	2784s	217
H→M	ViT-B/16 [ 9 ]	57.5	1211s	266
	DALL-V [ 34 ]	46.8	18368s	291

Table 6: Efficiency comparison of VUDA methods.

Figure 2: Visualization of LMFT on four videos (two left, two right). Each video has three rows: original frames, motion differences, and LMFT processed frames. LMFT selects action-relevant patches, dropping static or low-motion background patches for effective DA.

Confidence Threshold $\gamma_{c}$	Accuracy (%)
Confidence Threshold $\gamma_{c}$	M $\rightarrow$ H	H $\rightarrow$ M
0.0	68.8	54.3
0.2	67.5	57.0
0.4	68.8	52.7
0.6	72.5	57.7
0.8	74.2	60.0

Table 7: Impact of threshold

\gamma_{c}

in confidence-based filtering.

	Model	$N_{t}=2$	$N_{t}=4$	$N_{t}=6$	$N_{t}=8$
M→H	ViT-B/16 [ 9 ]	64.2	69.2	70.4	72.1
	Random	63.3	67.5	70.4	71.7
	UNITE [ 22 ]	63.3	70.4	68.8	71.7
	Ours	65.4	71.3	73.3	74.2
H→M	ViT-B/16 [ 9 ]	48.0	50.0	53.5	57.5
	Random	46.7	52.0	52.5	56.5

Table 8: Impact of different temporal resolutions on VUDA.

Reward hyperparameter $\lambda_{\mathcal{L}}$	Accuracy (%)
Reward hyperparameter $\lambda_{\mathcal{L}}$	M $\rightarrow$ H	H $\rightarrow$ M
0.1	73.3	56.7
1.0	73.8	57.0
10.0	74.2	60.0
100.0	72.8	58.5

Table 9: Effect of varying

\lambda_{\mathcal{L}}

in Eq. 7 .

M→H	Gumbel-Softmax	1.31	34.58	3275	73.3
	Method	Memory (GiB) ↓		Time ↓	Accuracy ↑
	Method	Min	Max	Time ↓	Accuracy ↑
M→H	LMFT(Ours)	1.31	30.03	2784	74.2
H→M	Gumbel-Softmax	1.31	34.58	1130	58.8
H→M	LMFT(Ours)	1.31	28.96	890	60.0

Table 10: Gumbel-Softmax vs. RL in LMFT.

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 179 papers

Return of Frustratingly Easy Unsupervised Video Domain Adaptation
May 19, 2026Pengfei Wei, Yiqun Sun, Zhiqiang Xu +2Unsupervised Domain Adaptation Video Action Recognition
1Magellan Technology Research Institute (MTRI), Japan · 2Mohamed bin Zayed University of Artificial Intelligence, UAE · 3Nanyang Technological University, Singapore.
Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation
Aug 29, 2026Lee En-Yi Hannah, Haozhi Cao, Yuecong XuTemporal Domain Adaptation Unsupervised Domain Adaptation
National University of Singapore · The University of Tokyo
Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts
Jul 27, 2026André Sacilotti, Samuel Felipe dos Santos, Jurandy AlmeidaSource-Free Domain Adaptation Test-Time Adaptation
University of São Paulo São Carlos, São Paulo, Brazil · Federal University of São Carlos Sorocaba, São Paulo, Brazil

May 19, 2026J/K move · Enter open · S save

Return of Frustratingly Easy Unsupervised Video Domain Adaptation

Pengfei Wei, Yiqun Sun, Zhiqiang Xu, Yiping Ke+1

1Magellan Technology Research Institute (MTRI), Japan · 2Mohamed bin Zayed University of Artificial Intelligence, UAE · 3Nanyang Technological University, Singapore.

Unsupervised Domain Adaptation Video Action Recognition

Open paper AskPDF arXiv

Unsupervised video domain adaptation (UVDA) is a practical but under-explored problem. In this paper, we propose a frustratingly easy UVDA method, called MetaTrans. Specifically, MetaTrans adopts a concise learning objective that contains only two fundamental loss terms. Despite the simplicity of the learning objective, MetaTrans embodies an advanced UVDA idea, that is, handling the spatial and temporal divergence of cross-domain videos separately, through a subtle model architecture design. By implementing a temporal-static subtraction module, MetaTrans effectively removes spatial and temporal divergence. Extensive empirical evaluations, particularly on various cross-domain action recognition tasks, show substantial absolute adaptation performance enhancement and significantly superior relative performance gain compared with state-of-the-art UVDA baselines.

Abstract

Unsupervised Domain Adaptation

Explore similar work

Return of Frustratingly Easy Unsupervised Video Domain Adaptation

Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation

Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts