Video Representation Learning

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 87

All topics

The TIME Machine: On The Power of Motion for Efficient Perception
May 21, 2026Mantas Skackauskas, Xinyue Hao, Laura Sevilla-LaraRepresentation Learning Video Representation Learning
School of Informatics University of Edinburgh
Seeing Through Fog: Towards Fog-Invariant Action Recognition
May 20, 2026Enqi Liu, Liyuan Pan, Zhi Gao +2Video Action Recognition Video Representation Learning
Beijing Institute of Technology, Beijing, China · Beijing Institute for General Artificial Intelligence, Beijing, China · Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China
Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
May 18, 2026Svetlana Orlova, Niccolò Cavagnero, Gijs DubbelmanStreaming Video Understanding Vision Foundation Model Adaptation
Eindhoven University of Technology
Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives
May 16, 2026Santosh PremiDisentangled Representation Learning Latent Dynamics Modeling
Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence
May 15, 2026Ali J Alrasheed, Aryan Yazdan Parast, Basim Azam +2Video Representation Learning Video Prediction
The University of Melbourne · Monash University
STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Contrastive Learning under Noisy Temporal Self-Supervision for Colonoscopy Videos
May 12, 2026Luca Parolari, Pietro Gori, Lamberto Ballan +2Contrastive Learning Endoscopy
Department of Mathematics, University of Padova, Padova, Italy · LTCI, Telecom Paris, Institut Polytechnique de Paris, Palaiseau, France · Cosmo Intelligent Medical Devices, Dublin, Ireland
Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation
May 11, 2026Kedi Sun, Chaohui Dang, Yue Feng +3Surgical Video Understanding Video Representation Learning
School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK · School of Computer Science, University of Birmingham, Birmingham, UK · Department of Applied Health Sciences, University of Birmingham, Birmingham, UK
Compressed Video Aggregator: Content-driven Module for Efficient Micro-Video Recommendation
May 9, 2026Yang Xiao, Huiyuan Chen, Kaiyuan Deng +5Video Representation Learning Embedding Compression
University of Tulsa · Amazon · University of Arizona +1
LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute
May 7, 2026Ali Salamatian, Anthony Fuller, Pritam Sarkar +3Efficient ViTs Video Action Recognition
University of British Columbia · Carleton University · Vector Institute
HumanNet: Scaling Human-centric Video Learning to One Million Hours
May 7, 2026Yufan Deng, Daquan ZhouHuman Activity Recognition Cross-Embodiment Robot Learning
DAGroup · SimpleSilicon Innovation Team Peking University
Video Generation with Predictive Latents
May 4, 2026Yian Zhao, Feng Wang, Qiushan Guo +4Variational Autoencoders Predictive Coding
ByteDance Seed · Tsinghua University · Peking University
TRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning
May 3, 2026Pritam Mishra, Coloma Ballester, Dimosthenis KaratzasReinforcement Learning Video Summarization
Universitat Pompeu Fabra, Barcelona, Spain · Universitat Autònoma de Barcelona, Barcelona, Spain
Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners
Apr 29, 2026Nikita Araslanov, Martin Sundermeyer, Hidenobu Matsuki +2Video Representation Learning 3D Scene Understanding
Google · TU Munich · Munich Center for Machine Learning
Micro-Expression-Aware Avatar Fingerprinting via Inter-Frame Feature Differencing
Apr 25, 2026Masoumeh Chapariniya, Jean-Marc Odobez, Volker Dellwo +1Biometric Identification Video Representation Learning
Department of Computational Linguistics, University of Zurich, Zurich, Switzerland · UZH Digital Society Initiative · Idiap Research Institute, Martigny, Switzerland +1
Efficient Logic Gate Networks for Video Copy Detection
Apr 23, 2026Katarzyna FojcikEfficient Neural Network Inference Video Representation Learning
Department of Artificial Intelligence, Wroclaw University of Science and Technology, Wroclaw, Poland
Temporal Prototyping and Hierarchical Alignment for Unsupervised Video-based Visible-Infrared Person Re-Identification
Apr 23, 2026Zhiyong Li, Wei Jiang, Haojie Liu +3Prototype Learning Video Representation Learning
College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · School of Computer Science and Technology, Zhejiang University of Water Resources and Electric Power, Hangzhou, 310018, China
Motion-Guided Semantic Alignment with Negative Prompts for Zero-Shot Video Action Recognition
Apr 18, 2026Yiming Wang, Frederick W. B. Li, Jingyun WangRepresentation Learning Video Action Recognition
Department of Computer Science, Durham University
CollideNet: Hierarchical Multi-scale Video Representation Learning with Disentanglement for Time-To-Collision Forecasting
Apr 17, 2026Nishq Poorav Desai, Ali Etemad, Michael GreenspanCollision Avoidance Video Representation Learning
Queen’s University, Canada
GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models
Apr 12, 2026Nicolae Cudlenco, Mihai Masala, Marius LeordeanuSynthetic Data Generation Video Representation Learning
Institute of Mathematics of the Romanian Academy, Bucharest, Romania · National University of Science and Technology Politehnica Bucharest, Romania · Büchi Labortechnik AG, Flawil, Switzerland
LRConv-NeRV: Low Rank Convolution for Efficient Neural Video Compression
Mar 18, 2026Tamer ShanablehVideo Representation Learning Neural Video Compression
American University of Sharjah, UAE, Department of Computer Science and Engineering
Xray-Visual Models: Scaling Vision models on Industry Scale Data
Feb 18, 2026Shlok Mishra, Tsung-Yu Lin, Linda Wang +24Vision Foundation Models Multimodal Pretraining
Meta-AI · MIT (work done while at Meta AI) · University of Maryland (work done while at Meta AI)
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Feb 5, 2026Jinlin Wu, Felix Holm, Chuxi Chen +18Joint-Embedding Predictive Architecture Surgical Video Understanding
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Jan 29, 2026Linhan Wang, Zichong Yang, Chen Bai +6End-to-End Autonomous Driving Joint-Embedding Predictive Architecture
Virginia Tech · Purdue University · XPENG Motors +1
FedVideoMAE: Efficient Federated Video Moderation with Differential Privacy and Secure Aggregation
Dec 21, 2025Ziyuan Tao, Chuanzhi Xu, Sandaru Jayawardana +4Federated Learning Aggregation Video Content Moderation
School of Computing, Macquarie University, Sydney, Australia · School of Computer Science, The University of Sydney, Sydney, Australia · School of Electrical and Computer Engineering, The University of Sydney, Sydney, Australia
Structured-Noise Masked Modeling for Video, Audio and Beyond
Mar 20, 2025Aritra Bhowmik, Carlos Hinojosa, Fida Mohammad Thoker +2Audio Representation Learning Video Representation Learning
University of Amsterdam · King Abdullah University of Science and Technology
What Moves? Localized Motion Representations for Compositional Scene Control
Date pendingFrank Fundel, Malek Ben Alaya, Thomas Ressler-Antal +2Representation Learning Object-Centric Representation Learning

May 21, 2026, cs.CVJ/K move · Enter open · S save

The TIME Machine: On The Power of Motion for Efficient Perception

Mantas Skackauskas, Xinyue Hao, Laura Sevilla-Lara

School of Informatics University of Edinburgh

Representation Learning Video Representation Learning Motion Representation Learning Self-Supervised Visual Representation Learning Temporal Video Understanding

Open paper AskPDF arXiv

Video representation learning has seen tremendous progress in recent years. This has been driven by many factors, including the scale of training and the success of self-supervised models trained on next-frame prediction. While these factors have pushed the boundaries of what video models can do, they also introduce their own set of limitations. First, scaling video models can reach prohibitive costs, with recent models needing hundreds of years of video to be trained. Second, learning to predict the next frame (or embedding of the frame) naturally focuses on spatial information and as a result, video models still struggle with temporal understanding. In this paper we propose a novel approach that uses motion as a modality to alleviate both of these core issues. Given the motion in a video in the form of point tracks, we mask some of the tracks and use a masked autoencoder to reconstruct the missing tracks. This allows us to learn a representation in a self-supervised manner which we call TIME (Temporally-Informed Motion Embedding), and is focused on capturing temporal information. Also, as motion is inherently appearance-invariant, TIME needs far fewer examples to generalize well. As a result, without bells and whistles, on temporal tasks TIME performs on par with state-of-the-art models, using up to 4 orders of magnitude less training data. For general tasks, TIME can be used in combination with existing representations, and we observe that it leads to a significant improvement for V-JEPA 2, RVM and VideoMAE on standard benchmarks such as SSV2, EgoExo4D and Diving48. These results point to a new promising video paradigm for both more temporally-aware as well as more scalable models.

Momentum

Related topics

Latest papers 87