It's a Matter of Time: Three Lessons on Long-Term Motion for Perception

Willem Davison; Xinyue Hao; Laura Sevilla-Lara

cs.CVFeb 16, 2026

It's a Matter of Time: Three Lessons on Long-Term Motion for Perception

Authors: Willem Davison, Xinyue Hao, Laura Sevilla-Lara

Organizations: University of Edinburgh

Abstract

Temporal information has long been considered to be essential for perception. While there is extensive research on the role of image information for perceptual tasks, the role of the temporal dimension remains less well understood: What can we learn about the world from long-term motion information? What properties does long-term motion information have for visual learning? We leverage recent success in point-track estimation, which offers an excellent opportunity to learn temporal representations and experiment on a variety of perceptual tasks. We draw 3 clear lessons: 1) Long-term motion representations contain information to understand actions, but also objects, materials, and spatial information, often even better than images. 2) Long-term motion representations generalize far better than image representations in low-data settings and in zero-shot tasks. 3) The very low dimensionality of motion information makes motion representations a better trade-off between GFLOPs and accuracy than standard video representations, and used together they achieve higher performance than video representations alone. We hope these insights will pave the way for the design of future models that leverage the power of long-term motion information for perception.

Figures & tables

Figure 2: MovT Architecture. Point-Tracks are factorized into motion and positional components for embedding. The resulting embeddings are fused for the final spatial transformer layer.

Figure 3: Lesson 1: Motion representations can solve a variety of tasks with high accuracy, comparable or better than video representations. We display the task-relevant performance metric in both the point-track and pixel input studies. Up and down arrows

(\uparrow/\downarrow)

signify whether a higher or lower performance metric is better. The red line on RAVDESS MovT at 31% marks the performance of our baseline experiment, where the facial landmark points are input to MovT for training.

Figure 4: Lesson 1: Many classes can be solved with motion information alone, including examples where image representations perform poorly. Scatter plots displaying a comparison of the per-class accuracy of our MovT and PixT models in three classification tasks.

Figure 5: Performance impact of temporal cropping SSV2 videos.

Figure 6: Lesson 2: Motion models have stronger generalization abilities in low-data settings. We test the effect of reducing the size of the training set progressively, and measure the resulting accuracy for MovT and PixT . In both datasets, the MovT loses less accuracy than PixT model, showing that motion representations indeed generalize better than image representations.

Figure 7: Lesson 2: Motion models have stronger generalization abilities in low-data settings. We test the effect of reducing the training data on the Jester dataset on a per-instance basis. We reduce the spatial fidelity of the input instances by randomly downsampling pixels and point-tracks.

Panel of Figure 8 — Figure 8: Lesson 2: Motion models have stronger generalization abilities in zero-shot tasks. Left: t-SNE visualization of Jester classes using the MovT trained on a different dataset, SSV2. We observe that classes are extremely well clustered using the MovT representation, even for a zero-shot task on an unseen dataset. Right: Loss of accuracy on the zero-shot task of training on SSV2 and testing on Jester.

Models	Jester			SSV2 (temporal)			VB100
Models	Frames	Top-1 Accuracy	GFLOPs	Frames	Top-1 Accuracy	GFLOPs	Frames	Top-1 Accuracy	GFLOPs
VideoMAE-small	1	30.49%	3.19	1	21.15%	3.19	1	5.11%	3.19
+ MovT	32	74.53% (+ 44.04%)	3.64 (+ 0.45)	32	54.68% (+ 33.53%)	3.71 (+ 0.52)	90	17.57% (+ 12.46%)	7.25 (+ 4.06)
VideoMAE-small	2	75.10%	3.24	2	49.65%	3.24	2	29.13%	3.24
+ MovT	32	87.47% (+ 12.37%)	3.69 (+ 0.45)	32	64.00% (+ 14.35%)	3.76 (+ 0.52)	90	29.88% (+ 0.75%)	7.30 (+ 4.06)
VideoMAE-small	4	90.64%	6.49	4	60.91%	6.49	4	33.18%	6.49

Table 1: Lesson 3: Motion representations can improve accuracy by a large margin while adding a low computational cost. Results of the fusion model for VideoMAE and MovT on Jester, SSV2, and VB100 datasets. Adding a motion representation improves the accuracy of VideoMAE in most settings by a fairly wide margin of up to 44%. This particular improvement comes at a moderate cost of 0.45 GFLOPs. The improvement is most noticeable in the case of action recognition classes.

Figure 9: Lesson 3: Motion representations can improve accuracy by a large margin while adding a low computational cost. Performance gain from incorporating MovT into VideoMAE. We can easily observe that MovT offers the best trade-off between computation and accuracy. We also observe that the combined model is the Pareto-front, showing optimality with respect to using only VideoMAE.

Figure 10: Visualization of point-tracks in the Jester dataset. For each video example, the left part shows all point-tracks, where colors transition from blue to red to indicate increasing importance scores. The right part highlights the top 10% most important tracks. As shown, the most important point-tracks are mainly located on the skeleton-related regions.

Figure 11: Point-track importance score histogram.

Figure 12: Comparison of different fusion techniques on the Jester dataset.

Figure 13: MovT performance on Jester above given confidence thresholds. Over a threshold of 0.978 confidence, we achieve 50% dataset coverage at 96.0% accuracy. Over a threshold of 0.743 confidence, we achieve 80% dataset coverage at 87.6% accuracy.

Model	GFLOPs	Parameters	Training Time
MovT	0.52	4.56M	50 mins
PixT	0.51	4.41M	45 mins
VideoMAE		16.63M	330 mins

Table 2: A comparison of the model complexities and training times for our different models (trained on the Jester dataset). VideoMAE is trained on full-scale images, while MovT and PixT are trained on point-tracks and scaled-down images. *VideoMAE GFLOPs varies, see Table 1 in our main paper submission.

	4 Frames		16 Frames
Model	GFLOPs ( $\downarrow$ )	Acc. ( $\uparrow$ )	GFLOPs ( $\downarrow$ )	Acc. ( $\uparrow$ )
MovT	0.30	70.86%	0.39	77.56%
VideoMAE-s	6.49 ( $22\times$ )	90.64%	25.95 ( $67\times$ )	95.85%

Table 15

Motion Representation Learning

Motion representation learning develops structured encodings of movement for recognition, prediction, generation, and control.

See all 152 papers

The TIME Machine: On The Power of Motion for Efficient Perception
May 21, 2026Mantas Skackauskas, Xinyue Hao, Laura Sevilla-LaraRepresentation Learning Video Representation Learning
School of Informatics University of Edinburgh
Learning Long-term Motion Embeddings for Efficient Kinematics Generation
Apr 13, 2026Nick Stracke, Kolja Bauer, Stefan Andreas Baumann +3Motion Representation Learning Human Motion Generation
Munich Center for Machine Learning · Apple
PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning
Mar 27, 2026Shaoxuan Li, Zhixuan Zhao, Hanze Deng +9Video Reasoning Temporal Video Understanding
Tsinghua University · Nanyang Technological University · University of Washington

May 21, 2026J/K move · Enter open · S save

The TIME Machine: On The Power of Motion for Efficient Perception

Mantas Skackauskas, Xinyue Hao, Laura Sevilla-Lara

School of Informatics University of Edinburgh

Representation Learning Video Representation Learning Motion Representation Learning Self-Supervised Visual Representation Learning Temporal Video Understanding Temporal Representation Learning

Open paper AskPDF arXiv

Video representation learning has seen tremendous progress in recent years. This has been driven by many factors, including the scale of training and the success of self-supervised models trained on next-frame prediction. While these factors have pushed the boundaries of what video models can do, they also introduce their own set of limitations. First, scaling video models can reach prohibitive costs, with recent models needing hundreds of years of video to be trained. Second, learning to predict the next frame (or embedding of the frame) naturally focuses on spatial information and as a result, video models still struggle with temporal understanding. In this paper we propose a novel approach that uses motion as a modality to alleviate both of these core issues. Given the motion in a video in the form of point tracks, we mask some of the tracks and use a masked autoencoder to reconstruct the missing tracks. This allows us to learn a representation in a self-supervised manner which we call TIME (Temporally-Informed Motion Embedding), and is focused on capturing temporal information. Also, as motion is inherently appearance-invariant, TIME needs far fewer examples to generalize well. As a result, without bells and whistles, on temporal tasks TIME performs on par with state-of-the-art models, using up to 4 orders of magnitude less training data. For general tasks, TIME can be used in combination with existing representations, and we observe that it leads to a significant improvement for V-JEPA 2, RVM and VideoMAE on standard benchmarks such as SSV2, EgoExo4D and Diving48. These results point to a new promising video paradigm for both more temporally-aware as well as more scalable models.

Abstract

Motion Representation Learning

Explore similar work

The TIME Machine: On The Power of Motion for Efficient Perception

Learning Long-term Motion Embeddings for Efficient Kinematics Generation

PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning