TrackMAE: Video Representation Learning via Track Mask and Predict

Renaud Vandeghen; Fida Mohammad Thoker; Marc Van Droogenbroeck; Bernard Ghanem

cs.CVMar 28, 2026

TrackMAE: Video Representation Learning via Track Mask and Predict

Authors: Renaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem

Organizations: University of Li`ege · KAUST

Abstract

Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos, generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction spaces by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms state-of-the-art video self-supervised learning baselines, learning more discriminative and generalizable representations. Code available at https://github.com/rvandeghen/TrackMAE

Figures & tables

Figure 1: TrackMAE improves masked video modeling by jointly predicting spatial features and motion trajectories in a mask-and-predict fashion.

Figure 2: Overview of TrackMAE. In the lower branch, a video clip $\mathbf{V}$ is first patchified and masked. The visible tokens are fed to a ViT encoder $\Phi$ . Then the decoder $\Psi_{spatial}$ aims to reconstruct spatial features based on the encoder output. In the upper branch, the input video clip is processed by a CoTracker3 module, extracting sparse point trajectories. The encoder output is then passed to a second decoder $\Psi_{Motion}$ , which aims to predict the extracted trajectories. The training objective combines both motion and spatial reconstruction.

Figure 3: Masking comparison. We show how our motion-based tube masking compares to random tube masking. By explicitly sampling visible tokens, our motion-based sampling distribution ensures that visible tokens cover both motion and static regions. In the motion-based tube sampling, red squares are sampled from the high-motion and blue squares from the low-motion bin.

Method	Target	Spatial-centric		Motion-centric
Method	Target	K400	HMDB	SSv2	GYM
Pixel Reconstruction
VideoMAE [ 49 ]	Pixel	20.7	37.7	17.5	23.9
MVD [ 51 ]	Pixel	18.7	28.6	12.2	22.7
MGMAE [ 23 ]	Pixel	24.9	41.3	16.8	26.1
EVEREST [ 25 ]	Pixel	14.1	30.3	14.5	23.3

Table 1: Linear probing comparison. We evaluate TrackMAE on different spatial- and motion-centric benchmarks. For both pixel and feature reconstruction, our method is on par with previous methods for spatial-centric tasks. For motion-centric tasks, our method largely improves previous methods. We report the Top-1% accuracies of ViT-B models pretrained on K400, and highlight in bold best and underline second best results.

Method	Backbone	Targets	SSv2 Top-1	K400 Top-1
VideoMAE [ 49 ]	ViT-B	Pixel	68.5	80.0
OmniMAE [ 17 ]	ViT-B	Pixel	69.0	80.8
MGM [ 14 ]	ViT-B	Pixel	71.1	80.8
MGMAE [ 23 ]	ViT-B	Pixel	68.9	81.2
TrackMAE (ours)	ViT-B	Pixel	70.1	80.8
MME [ 45 ]	ViT-B	HOG	70.5	81.5

Table 2: Full finetuning comparison. We evaluate TrackMAE finetuned on SSv2 and K400 after pretraining the models on K400, outperforming all prior works. We highlight in bold best and underline second best results.

\dagger

means we pretrained on K700.

Reconstruction Targets	$\textrm{K400}_{\textrm{s}}$	$\textrm{SSv2}_{\textrm{s}}$
Trajectory only	46.5	53.1
Pixels	46.0	52.2
Pixels + Trajectory	48.9	55.7
CLIP	52.7	57.1
CLIP + Trajectory	55.8	61.1

Table 3: Reconstruction targets. Trajectory reconstruction provides a strong supervisory signal to learn useful video representations and complements both pixels and CLIP reconstruction, consistently improving the downstream results.

Target	Masking	$\textrm{K400}_{\textrm{s}}$	$\textrm{SSv2}_{\textrm{s}}$
Pixel	Tube	46.0	52.2
Pixel	Motion-aware	46.6	52.6
Pixel + Trajectory	Tube	48.9	55.7
Pixel + Trajectory	Motion-aware	49.4	56.2

Table 4: Masking strategy. Replacing random tube masking with our motion-based masking consistently improves the results for both pixels and pixels + trajectory reconstruction.

Grid Size	Upsampling	$\textrm{K400}_{\textrm{s}}$	$\textrm{SSv2}_{\textrm{s}}$
$14\times 14$	None	48.9	55.7
$28\times 28$	None	49.5	56.7
$56\times 56$	None	50.0	57.0
$14\times 14$	$14\rightarrow 28$ ( $\upsilon{=}2$ )	50.6	57.6
$14\times 14$	$14\rightarrow 56$ ( $\upsilon{=}4$ )	50.4	57.4

Table 5: Impact of dense tracking. Increasing the tracker grid size consistently improves the results, but at a higher computational cost. Upsampling the tracker’s prediction from

14\times 14

28\times 28

yields better performance without any added cost.

Table 9

Method	Domain shift		Sample efficiency ( $10^{3}$ )		Action granularity		Task shift
Method	SSv2	Gym99	UCF	GYM	FX-S1	UB-S1	UCF-RC $\downarrow$	Charades
Pixel Reconstruction
VideoMAE [ 49 ]	68.6	86.6	74.6	25.9	42.8	65.3	0.172	17.8
MVD [ 51 ]	70.0	82.5	67.1	17.5	31.3	50.5	0.184	16.1
MGMAE [ 23 ]	68.9	87.2	77.2	24.1	33.7	79.5	0.181	17.9
MGM [ 14 ]	71.1	89.1	78.4	26.4	38.6	86.9	0.152	22.5

Table 8: Comparison on SEVERE generalization benchmark [ 46 ] . We compare prior video SSL methods on four generalization factors of SEVERE benchmark spanning a total of eight downstream settings. TrackMAE delivers consistently strong or superior performance across these diverse settings in both pixel and feature reconstruction modes, indicating that trajectory-guided masked pretraining improves robustness and generalization of learned video representations.

				SSv2 Pretraining	K400 Pretraining
Method	Backbone	Targets	Epochs	SSv2 Top-1	SSv2 Top-1	K400 Top-1
Supervised
Mformer [ 37 ]	Mformer-B	-	-	-	66.7	79.7
VideoSwin [ 34 ]	Swin-B	-	-	-	69.6	80.6
TimeSformer [ 4 ]	ViT-B	-	-	-	59.5	80.7
MViTv1 [ 15 ]	MViTv1-B	-	-	-	67.7	80.2

Table 9: Detailed SOTA comparison of masked video modeling methods on Something-Something V2 and Kinetics-400 for full finetuning action recognition. Our TrackMAE outperforms many supervised approaches and achieves the best performance masked video modeling methods with similar pretraining setups.

\dagger

means we pretrained on K700.

Table 10: Additional ablations on our proposed TrackMAE. The default setting uses pixel reconstruction and motion prediction with a grid size of 14, masking ratio of 90%,

\lambda=1

, and two separate decoders.

Figure 4: Training evolution. We report the Top-1 Accuracy for K400 and SSv2 finetuning at different pretraining epochs. Our model trained with both CLIP and trajectory reconstructions consistently outperforms the CLIP reconstruction only.

Figure 5: Sampling strategies. We show how we can use the motion information to create different sampling distribution.

Figure 6: Masking strategies. We show how we can use our sampling distribution to create different masking maps. Red squares are high-motion visible tubes, blue squares are low-motion visible tokens. For the ”exclude topk” strategy, we also show orange the tokens excluded from being sampled.

Shared
Optimizer	AdamW
Base learning rate	$1.5\times 10^{-4}$
Weight decay	0.05
Momentum (Betas)	$\beta_{1}{=}0.9,\ \beta_{2}{=}0.95$
Batch size	512
LR schedule	cosine decay

Table 11: Pretraining configuration.

config	K400	HMDB	SSv2	GYM
optimizer	AdamW
base learning rate	1 $\times$ 10 -3
weight decay	0.05
optimizer momentum	$\beta_{1},\beta_{2}=0.9,0.999$
layer-wise lr decay	0.75
batch size	128

Table 12: Linear probing configuration.

Parameter	Value
Optimizer	AdamW
Base learning rate	$1.0\times 10^{-3}$
Weight decay	0.05
Momentum (Betas)	$\beta_{1}=0.9,\ \beta_{2}=0.999$
Layer-wise LR decay	0.75
LR schedule	cosine decay

Table 13: Full finetuning configuration.

Dataset	Abbrev.	#Classes	#Train	#Test
Kinetics - 400	K400	400	240K	19K
UCF - 101	UCF	101	9.5K	3.8K
HMDB - 51	HMDB	51	4.8K	2K
Something - Something V2	SSv2	174	169K	24.8K
FineGYM	GYM	99	20.5K	8.5K

Table 14: Datasets details. Splits of datasets used in full finetuning and linear probing.

Dataset	Experiment	Setup Group	Task	#Classes	Finetune	Test	Metric
FineGym [ 42 ]	Gym99	Full	Action Class.	99	20,484	8,521	Top-1 Acc.
UCF 101 [ 44 ]	UCF ( $10^{3}$ )	Sample Efficiency	Action Class.	101	1,000	3,783	Top-1 Acc.
FineGym [ 42 ]	Gym ( $10^{3}$ )	Sample Efficiency	Action Class.	99	1,000	8,521	Top-1 Acc.
FineGym [ 42 ]	FX - S1	Action Granularity	Action Class.	11	1,882	777	Mean-per-class
FineGym [ 42 ]	UB - S1	Action Granularity	Action Class.	15	3,511	1,471	Mean-per-class
UCFRep [ 57 ]	UCF - RC	Task Shift	Repetition Counting	–	421	105	Mean Error

Table 15: SEVERE benchmark. Subsets, protocols, and metrics following [ 46 ] .

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 143 papers

The TIME Machine: On The Power of Motion for Efficient Perception
May 21, 2026Mantas Skackauskas, Xinyue Hao, Laura Sevilla-LaraRepresentation Learning Video Representation Learning
School of Informatics University of Edinburgh
Image Classifiers are Efficient Self-Supervised Video Representation Learners
Sep 30, 2026Owais Iqbal, Sudipta Sarkar, Shyam Marjit +3Self-Supervised Pre-Training Video Representation Learning
Indian Institute of Technology Kharagpur, India · Indian Institute of Science Bangalore, India · École de technologie supérieure Montreal, Canada
Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning
Jun 8, 2026Qinwu XuVideo Understanding Video Representation Learning
∗This work was independently developed during Early Feb –Mid April 2026.

May 21, 2026J/K move · Enter open · S save

The TIME Machine: On The Power of Motion for Efficient Perception

Mantas Skackauskas, Xinyue Hao, Laura Sevilla-Lara

School of Informatics University of Edinburgh

Representation Learning Video Representation Learning Motion Representation Learning Self-Supervised Visual Representation Learning Temporal Video Understanding Temporal Representation Learning

Open paper AskPDF arXiv

Video representation learning has seen tremendous progress in recent years. This has been driven by many factors, including the scale of training and the success of self-supervised models trained on next-frame prediction. While these factors have pushed the boundaries of what video models can do, they also introduce their own set of limitations. First, scaling video models can reach prohibitive costs, with recent models needing hundreds of years of video to be trained. Second, learning to predict the next frame (or embedding of the frame) naturally focuses on spatial information and as a result, video models still struggle with temporal understanding. In this paper we propose a novel approach that uses motion as a modality to alleviate both of these core issues. Given the motion in a video in the form of point tracks, we mask some of the tracks and use a masked autoencoder to reconstruct the missing tracks. This allows us to learn a representation in a self-supervised manner which we call TIME (Temporally-Informed Motion Embedding), and is focused on capturing temporal information. Also, as motion is inherently appearance-invariant, TIME needs far fewer examples to generalize well. As a result, without bells and whistles, on temporal tasks TIME performs on par with state-of-the-art models, using up to 4 orders of magnitude less training data. For general tasks, TIME can be used in combination with existing representations, and we observe that it leads to a significant improvement for V-JEPA 2, RVM and VideoMAE on standard benchmarks such as SSV2, EgoExo4D and Diving48. These results point to a new promising video paradigm for both more temporally-aware as well as more scalable models.

Abstract

Video Representation Learning

Explore similar work

The TIME Machine: On The Power of Motion for Efficient Perception

Image Classifiers are Efficient Self-Supervised Video Representation Learners

Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning