From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning

Yang Liu; Qianqian Xu; Peisong Wen; Siran Dai; Xilin Zhao; Qingming Huang

cs.CVMar 27, 2026

From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning

Authors: Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, Xilin Zhao, Qingming Huang

Organizations: School of Computer Science and Technology, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence · Institute of Information Engineering, Chinese Academy of Sciences ·

Abstract

Recent studies have made notable progress in video representation learning by transferring image-pretrained models to video tasks, typically with complex temporal modules and video fine-tuning. However, fine-tuning heavy modules may compromise inter-video semantic separability, i.e., the essential ability to distinguish objects across videos. While reducing the tunable parameters hinders their intra-video temporal consistency, which is required for stable representations of the same object within a video. This dilemma indicates a potential trade-off between the intra-video temporal consistency and inter-video semantic separability during image-to-video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks with only five epochs of self-supervised training. The code is available at https://github.com/yafeng19/Co-Settle.

Figures & tables

Figure 1: Comparison of video representation quality with recent visual representation learning models on the Kinetics-400 [ 55 ] validation set. Favorable video representations should exhibit strong intra-video temporal consistency (lower intra-video distance

D_{intra}

) and clear inter-video semantic separability (higher inter-video distance

D_{inter}

) jointly, yet the two objectives often compete since the two distances co-vary. Applying our method to image-pretrained models leads to consistent improvements on the margin of inter- and intra-video distance

D=D_{inter}-\gamma D_{intra}

(detailed in Sec. 5.3 ), indicating a better trade-off between the two properties, and therefore leading to improved performance on video downstream tasks.

Figure 2: Overview of our image-to-video transfer learning framework. Two frames are sampled from each video to construct a cyclic sequence. A frozen image-pretrained encoder extracts patch-level features, which are then mapped by a learnable projection layer. The projection layer is trained with a temporal cycle-consistency loss and a semantic separability constraint for representation adjustment, thereby promoting a better trade-off between intra-video temporal consistency and inter-video semantic separability.

Panel of Figure 3 — Figure 3: Left : Observations on shortcuts. Patches with the same color box denote correspondence. Middle : Overview of our PEA strategy. Right : Cycle-consistent accuracy and downstream performance dynamics during training with or without our PEA strategy on MAE encoder.

Type	Method	Backbone	Dataset	Epoch	VIP	DAVIS-2017			JHMDB
Type	Method	Backbone	Dataset	Epoch	mIoU	$\mathcal{J}\&\mathcal{F}_{\mathrm{m}}$	$\mathcal{J}_{\mathrm{m}}$	$\mathcal{F}_{\mathrm{m}}$	[email protected]	[email protected]
Video Pretrained	VideoMAE ‡ [ 101 ]	ViT-L/16	K400	1600	25.6	45.0	43.6	46.5	43.3	70.5
	MAE-ST † [ 33 ]	ViT-L/16	K400	1600	33.2	54.6	55.5	53.6	44.4	72.5
	DropMAE † [ 111 ]	ViT-B/16	K400	1600	31.1	53.4	51.8	55.0	42.3	69.2
	SiamMAE [ 39 ]	ViT-B/16	K400	400	36.1	60.9	59.4	62.4	47.0	74.9
	CropMAE † [ 30 ]	ViT-B/16	K400	400	33.0	57.8	56.9	58.7	45.3	73.3

Table 1: Comparison with image and video representation learning methods on three dense-level video downstream tasks. Results marked with † are reported from prior works; ‡ denotes evaluations based on the official pretrained weights. The best results are highlighted in bold . INet, K400, WIT, and LAION represent ImageNet-1k [ 25 ] , Kinetics-400 [ 55 ] , WIT-400M [ 82 ] , and LAION-400M [ 87 ] datasets.

Figure 4: Evaluation results on frame-level and video-level tasks based on four representative image models.

Features	BADJA
Features	$\delta^{seg}$	$\delta^{3px}$
DINO v2	62.73	8.85
+ Ours	70.52	8.86
Features	TAP-DAVIS
Features	$\delta_{avg}^{x}$	OA
DINO v2	64.68	81.98

Table 2: Results of integration into video dense tracking pipeline.

Method	Tunable Params $(\downarrow)$	Peak GPU Mem. $(\downarrow)$	Tuning GPU Hours $(\downarrow)$	VIP	DAVIS17	JHMDB
Method	Tunable Params $(\downarrow)$	Peak GPU Mem. $(\downarrow)$	Tuning GPU Hours $(\downarrow)$	$\mathcal{J}\&\mathcal{F}_{\mathrm{m}}$	mIoU	[email protected]
MAE	/	/	/	29.3	52.4	41.6
Full Fine-tuning	111.66 M	13.9 GB	20.1 $\times$ RTX4090	29.8	53.9	42.6
Partial Fine-tuning	7.09 M	9.2 GB	15.6 $\times$ RTX4090	29.2	52.7	41.3
I2V Adapter	14.22 M	20.4 GB	21.2 $\times$ RTX4090	29.5	53.0	41.5
Ours	0.59 M	4.8 GB	1.2 $\times$ RTX4090	33.8	59.6	48.4

Table 3: Efficiency and performance comparison with image-to-video transfer methods. † marks the costs from prior studies.

Type	Method	$D_{inter}$	$D_{intra}$	$D(\uparrow)$	Cyc. Acc. $(\uparrow)$
Video Pretrained	SiamMAE	0.5067	0.1330	0.4668	0.4100
	CropMAE	0.5216	0.1736	0.4695	0.6522
	RSP	0.4662	0.2130	0.4023	0.5990
Image Pretrained + Ours	MAE	0.3122	0.1131	0.2783	0.1366
	MAE + Ours	0.5073	0.1834	0.4523	0.7203
	I-JEPA	0.2572	0.1425	0.2145	0.1192

Table 4: Validation results on distance-based trade-off metrics.

Table 5: Ablation results on the components, structures, and training configurations of our method. The best and secondary best results are highlighted in bold and underlined . Default settings are marked with blue .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Descriptions
$\mathcal{D}$	Training dataset.
$H/W/C$	The height/width/channel dimension of a frame.
$T$	The number of frames in a video.
$\bm{V}$	A video containing $T$ frames.
$\bm{v}_{t}$	The frame at the moment $t$ in a video, $\bm{v}_{t}\in\mathbb{R}^{H\times W\times C}$ .
$\bm{v}_{t}(i)$	A frame patch of $\bm{v}_{t}$ , $\bm{v}_{t}\in\mathbb{R}^{H\times W\times C}$ .

Appendix

Table 6: A summary of key notations and descriptions used in the Method Section.

Notations	Descriptions
$d$	The embedding dimension of a frame patch.
$\lambda$	The strength of the constraint term.
$f(\cdot)$	The image-pretrained encoder, $f:\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{N\times d}$ .
$g(\cdot)$	The projection layer, $g:\mathbb{R}^{N\times d}\rightarrow\mathbb{R}^{N\times d}$ .
$\bm{z}_{i}$	The latent representation of an input patch.
$\bar{\bm{z}}_{i}$	The mean representation of video $\bm{V}_{i}$ , $\bar{\bm{z}}_{i}=\mathbb{E}_{\bm{z}\in f(\bm{V}_{i})}\left[\bm{z}\right]$ .

Appendix

Table 7: A summary of key notations and descriptions used in the Theoretical Analysis Section.

Panel of Figure 5 — Figure 5: Comparison of several categories of video representation learning methods with ours.

Table 8: Summary of hyperparameter settings used during training and evaluation.

Method	$\gamma$	Method	$\gamma$
MAE	0.1855	MoCov3	0.3321
MAE + Ours	0.3289	MoCov3 + Ours	0.3053
I-JEPA	0.2283	iBOT	0.2817
I-JEPA + Ours	0.2645	iBOT + Ours	0.3084
CLIP	0.3365	DINO	0.3378
CLIP + Ours	0.3876	DINO + Ours	0.3505

Appendix

Table 9: The scale factor

\gamma={D_{intra}^{ori}}/{D_{inter}^{ori}}

and the average scale factor for each model.

Model	Action Localization					Video Retrieval				Action Classification		Temporal Order Discrimination
	Breakfast					UCF101		HMDB51		SSV2		Chiral SSV2
	Edit	Acc	[email protected]	[email protected]	[email protected]	mAP	R@1	mAP	R@1	Acc@1	Acc@5	Acc
CLIP	53.8	40.1	52.9	46.0	33.8	45.9	90.8	25.5	70.1	34.6	67.8	79.1
CLIP +Ours	54.9	40.9	52.5	46.4	34.8	49.0	96.0	27.1	71.3	35.5	68.8	80.5
BLIP	57.3	47.3	55.6	49.4	36.7	54.4	96.4	29.4	73.3	39.9	72.4	81.8
BLIP +Ours	58.6	49.1	57.5	51.0	38.1	55.2	97.0	29.9	73.5	41.4	72.6	82.8

Appendix

Table 10: Evaluation results on frame-level and video-level tasks based on representative image models. The best results are marked in bold .

Panel of Figure 6 — Figure 6: Cycle-consistent accuracy and downstream performance during training with or without the PEA strategies across three tasks based on MAE ( line 1 ) and DINO ( line 2 ).

Type	Method	Backbone	DAVIS-2017
Type	Method	Backbone	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$
Dedicated VOS Systems	STCN [ 22 ]	ResNet50	85.4	82.2	88.6
	SwinB-AOT-L [ 116 ]	Swin-B	85.4	82.4	88.4
	SimVOS-B [ 112 ]	ViT-B/16	88.0	85.0	91.0
	Cutie-base [ 21 ]	ResNet50	87.9	84.6	91.1
Segmentation Foundation Models	SAM 2 [ 84 ]	Hiera-B+	90.2	87.0	93.4

Appendix

Table 11: Extended comparison with representative methods on the DAVIS-2017 validation set. Methods are grouped by their core settings for a broader reference.

Image Model	Method	VIP	DAVIS17	JHMDB
Image Model	Method	mIoU	$\mathcal{J}\&\mathcal{F}_{\mathrm{m}}$	[email protected]
MAE	Vanilla	29.3	52.4	41.6
	w/o PEA	16.2 -13.1	26.2 -26.2	38.5 -3.1
	w/ PEA	33.8 +4.5	59.6 +7.2	48.4 +6.8
I-JEPA	Vanilla	31.5	53.9	42.6
	w/o PEA	26.6 -4.9	49.5 -4.4	44.1 +1.5

Appendix

Table 12: Impact of Positional Encoding Augmentation (PEA) strategy on representation quality across three downstream tasks.

Figure 7: 3D bar charts for ablation results on interpolation ratio

\alpha

and regularization weight

\lambda

across three tasks using MAE.

Table 13: Sensitivity analysis on temporal interval

\delta

and softmax temporature

\tau

. Default settings are highlighted with blue .

Table 14: Robustness and generalization analysis of PEA strategy across DINO series features.

PEA	$L_{reg}$	DAVIS	VIP	JHMDB	$D_{inter}$	$D_{intra}$	$D~(\uparrow)$
DINO		63.2	39.1	44.4	0.5756	0.2144	0.5112
✗	KL	61.8	38.0	46.1	0.6241	0.2404	0.5520
✓	MSE	62.2	38.1	46.1	0.6142	0.2370	0.5431
✓	KL	64.2	39.8	46.2	0.6246	0.2316	0.5551

Appendix

Table 15: Ablation study on the choice of regularization loss (

L_{reg}

) using the DINO backbone.

Figure 8: Cross-frame correspondence learned with our method. Patches with the same color box represent correspondence.

Figure 9: Visualization comparison across three downstream tasks based on MAE.

Figure 10: Visualization comparison across three downstream tasks based on CLIP.

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 143 papers

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining
Sep 27, 2026Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang +3Efficient ViTs Video Representation Learning
National Tsing Hua University · Comfy Org Research · Karolinska Institutet +2
Image Classifiers are Efficient Self-Supervised Video Representation Learners
Sep 30, 2026Owais Iqbal, Sudipta Sarkar, Shyam Marjit +3Self-Supervised Pre-Training Video Representation Learning
Indian Institute of Technology Kharagpur, India · Indian Institute of Science Bangalore, India · École de technologie supérieure Montreal, Canada
Self-Supervised Learning of Structured Dynamics from Videos
Jul 23, 2026Lukas Knobel, Andrew Zisserman, Yuki M. AsanoVideo Representation Learning Self-Supervised Visual Representation Learning
Fundamental AI Lab, UTN · VGG, University of Oxford

Sep 27, 2026J/K move · Enter open · S save

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang, Fu-En Yang+2

National Tsing Hua University · Comfy Org Research · Karolinska Institutet+2

Efficient ViTs Video Representation Learning Motion Representation Learning

Open paper AskPDF arXiv

Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched

4 \times 6 = 24

architecture-objective study at roughly 170M ~ 190M encoder scale on

\sim

1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.

Abstract

Video Representation Learning

Explore similar work

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Image Classifiers are Efficient Self-Supervised Video Representation Learners

Self-Supervised Learning of Structured Dynamics from Videos