Video Representation Learning

Overview

Video representation learning studies how to encode frames and their evolution into features useful for tasks such as action recognition, dense tracking, retrieval, generation, and robot planning. Common approaches include masked reconstruction, contrastive learning, predictive latent objectives such as JEPA, explicit motion and trajectory cues, multimodal alignment, and transfer from image encoders. Evaluations span action datasets such as Kinetics and Something-Something, egocentric and tracking benchmarks such as Ego4D and TAP-Vid, and embodied settings such as NAVSIM and robotic manipulation. A central challenge is learning features that preserve semantic appearance while capturing temporal order, geometry, and action-relevant dynamics without prohibitive data and compute costs; standard recognition scores do not always reveal these capabilities.

Latest papers

151 papers14 papers in the last four weeks, down 33% on the four weeks before. 0.1% of all new papers.

Mar 20, 2025

Structured-Noise Masked Modeling for Video, Audio and Beyond

Masked modeling has emerged as a robust self-supervised learning framework. However, most methods rely on random masking, which disregards the structural properties of different data modalities. To align with the spatiotemporal and spectral characteristics of video and audio data, we introduce a structured noise-based masking approach. By filtering white noise into different color noise distributions, we generate structured masks that capture modality-specific patterns without requiring handcrafted heuristics or access to the data. Our approach enhances masked video and audio modeling frameworks without any additional computational cost. Experiments show that structured noise masking consistently outperforms random masking, underscoring the value of modality-aware masking strategies for representation learning.