Video Representation Learning

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 87

All topics

Self-Supervised Learning of Structured Dynamics from Videos
Jul 23, 2026Lukas Knobel, Andrew Zisserman, Yuki M. AsanoVideo Representation Learning Self-Supervised Visual Representation Learning
Fundamental AI Lab, UTN · VGG, University of Oxford
Sidewalk Moments: Are Richer Representations Always More Human-Aligned? Evidence from City-Walk Videos
Jul 23, 2026Liu Liu, Freya Huying Tan, Fábio DuarteEgocentric Video Understanding Video Representation Learning
Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA · City Form Lab, MIT · Senseable City Lab, MIT
Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition
Jul 22, 2026Jiajun Cheng, Sainan Liu, Subarna Tripathi +2Latent Action Learning Surgical Video Understanding
Arizona State University, Tempe, AZ, USA · Intel Corporation, USA · University of California, Merced, CA, USA
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
Jul 15, 2026Zhihao Xie, Junfeng Wu, Xinting Hu +2Video Diffusion Models Video Representation Learning
The Chinese University of Hong Kong, Shenzhen · Huazhong University of Science and Technology · University of Science and Technology of China +1
HyperGS: Fast and Generalizable Gaussian Video Representation
Jul 13, 2026Fatimah Zohra, Chen Zhao, Shuming Liu +23D Gaussian Splatting Video Representation Learning
1King Abdullah University of Science and Technology (KAUST) · 2Harbin Institute of Technology, Shenzhen
`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation
Jul 8, 2026Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew +1Video Object Segmentation Self-Supervised Learning
School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.
Gen4U: Unifying Video Generation and Understanding via Diffusion
Jul 7, 2026Michael King, Aravindh Mahendran, Matthew Koichi Grimes +5Video Diffusion Models Video Representation Learning
Google DeepMind
From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training
Jul 1, 2026Jinwen Wang, Youfang Lin, Xiaobo Hu +4Contrastive Learning Reinforcement Learning
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University, Beijing, China · Beijing Jiaotong University, Beijing, China
Local Motion Matters: A Deconstruct-Recompose Paradigm for Reinforcement Learning Pre-training from Videos
Jul 1, 2026Jinwen Wang, Youfang Lin, Xiaobo Hu +2Cross-Embodiment Robot Learning Robotic RL
Beijing Jiaotong University, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, China
Neural Voxel Dynamics: Learning Implicit 3D Physics via Volumetric Feature Advection
Jun 24, 2026Zican Wang, Niloy MitraVideo Representation Learning 3D Scene Representation
University College London · Adobe Research
P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture
Jun 22, 2026Felix Tristram, Stefano Gasperini, Benjamin Killeen +4Temporal Action Segmentation Video Understanding
Technical University of Munich Munich Center for Machine Learning · Carl Zeiss AG · Carl Zeiss AG[cs.CV]
LEViL: Label-Efficient Video Learning via Zero-Shot Distillation over VLM-Generated Pseudo-Label Spaces
Jun 19, 2026Aslı ÇelikVLM Distillation Video Action Recognition
Department of Electronics and Communications Engineering Kocaeli University Kocaeli, Türkiye
UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
Jun 18, 2026Wenhao Chi, Arkaprava Sinha, Dominick Reilly +2Cross-Modal Knowledge Distillation Multi-View Learning
University of North Carolina at Charlotte
DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval
Jun 17, 2026Kaleem Ullah, Altaf Hussain, Muhammad Munsif +1Cross-Modal Retrieval Video Representation Learning
Sejong University, Seoul 143-747, South Korea · KAIST InnoCORE PRISM-AI Center, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, Republic of Korea · Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea
Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins
Jun 15, 2026Yiqing Shen, Hao Ding, Mathias UnberathSurgical Video Understanding Video Representation Learning
Johns Hopkins University, Baltimore, MD, USA
You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences
Jun 14, 2026Ninad Daithankar, Alexi Gladstone, Yann LeCun +1Visual Representation Learning Video Representation Learning
1UIUC · 2New York University
Selective Synergistic Learning for Video Object-Centric Learning
Jun 14, 2026WonJun Moon, Jae-Pil HeoObject-Centric Representation Learning Video Representation Learning
KAIST, South Korea · † Work done at Sungkyunkwan University. · Sungkyunkwan University, South Korea
MVEB: Massive Video Embedding Benchmark
Jun 12, 2026Adnan El Assadi, Roman Solomatin, Isaac Chung +13Multimodal Embedding Video Representation Learning
1Harvard University · 2SaluteDevices · 3MIRAI +11
MLT-Dedup: Efficient Large-Scale Online Video Deduplication via Multi-Level Representations and Spatial-Temporal Matching
Jun 10, 2026David Yuchen Wang, Haoying Li, Hailun Xu +6Video Representation Learning
TikTok Singapore, Singapore · School of Computing National University of Singapore Singapore, Singapore · TikTok San Jose, CA, USA
Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning
Jun 8, 2026Qinwu XuVideo Understanding Video Representation Learning
∗This work was independently developed during Early Feb –Mid April 2026.
Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis
Jun 8, 2026Samuele Punzo, Niccolò Caselli, Ippokratis Pantelidis +3Vision Foundation Models Video Representation Learning
University of Amsterdam, Netherlands.
Self-supervised Learning Matters: A Simple Ensemble Solution for Micro-Gesture Recognition
Jun 8, 2026Tingyi Liu, Kun Li, Fei Wang +5Video Representation Learning Gesture Recognition
Hefei University of Technology, Hefei, China · United Arab Emirates University, Al Ain, United Arab Emirates · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China +5
What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
Jun 5, 2026Jewon Yeom, Hanseul Kim, Jeongjae Park +3Video Representation Learning Predictive Representation Learning
Graduate School of Data Science, Seoul National University
Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models
Jun 4, 2026Haibo Wang, Lifu Huang3D Spatial Reasoning Multimodal Large Language Models
University of California, Davis
The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show
Jun 3, 2026Parsa Esmati, Somjit Nath, Katja Hofmann +3Video Diffusion Models Video Representation Learning
University of Bristol · McGill University · Mila–Quebec AI Institute +2
TrAction: Action Recognition with Sparse Trajectories
Jun 2, 2026Jan F. Meier, Felix B. Mueller, Alexander Ecker +1Video Action Recognition Video Representation Learning
Institute of Computer Science and Campus Institute Data Science, University Göttingen, Germany · Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany
Paving the Way for Point Cloud Video Representation Learning Using A PDE Model
Jun 1, 2026Zhuoxu Huang, Zhenkun Fan, Jungong Han +1Point Cloud Learning Video Representation Learning
Department of Computer Science, Aberystwyth University, Aberystwyth SY23 3DB, U.K. · Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China · Department of Electrical Engineering, Surrey University, Surrey GU2 7XH, U.K.
CoSTL: Comprehensive Spatial-Temporal Representation Learning for Moment Retrieval and Highlight Detection
May 31, 2026Xin Dong, Wenjia Geng, Wenfeng Deng +1Temporal Video Grounding Video Representation Learning
Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory
VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer
May 27, 2026Rui Lin, Chuanming Wang, Huadong MaVLM Adaptation Video Representation Learning
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, China
Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models
May 27, 2026Haozhan Shen, Tiancheng Zhao, Kangjia Zhao +1Vision-Language Models Visual Spatial Reasoning
Zhejiang University · Om AI Research · Binjiang Institute of Zhejiang University

Jul 23, 2026, cs.CVJ/K move · Enter open · S save

Self-Supervised Learning of Structured Dynamics from Videos

Lukas Knobel, Andrew Zisserman, Yuki M. Asano

Fundamental AI Lab, UTN · VGG, University of Oxford

Video Representation Learning Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent or with unstructured, spatially dense transition tokens. Training combines self-supervised learning on real video with weak supervision of scene dynamics on synthetic Kubric data. We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features, and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision. These results suggest that pretrained image models can be readily repurposed into structured video-dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

Momentum

Related topics

Latest papers 87