Overview

Video understanding studies how systems interpret actions, objects, interactions, and events as they unfold, supporting tasks such as action recognition, temporal localization, video question answering, and summarization. Approaches range from spatiotemporal transformers and self-supervised motion learning to multimodal language models that use retrieval, memory, or visual tools to analyze long videos. Benchmarks test short clips, long-form and egocentric footage, procedural tasks, and audio-visual settings, but results can depend heavily on frame sampling, token budgets, and evaluation design. A central challenge is making models rely on temporal evidence and maintain coherent reasoning over long or multiple videos rather than exploiting static cues or struggling with context limits.

Latest papers

122 papers11 papers in the last four weeks, down 48% on the four weeks before. 0.1% of all new papers.

Dec 3, 2025

Benchmarking and Learning Object-Centric Procedural Understanding in Instructional Videos

Procedural activities are fundamentally driven by object state transitions, yet existing instructional video benchmarks remain action-centric and cannot evaluate whether models reason about how objects evolve toward task completion. In this work, we introduce ProcObject-10K, the first benchmark that jointly evaluates object-centric reasoning and temporal evidence grounding in instructional videos, across both egocentric and exocentric views. It comprises 10,522 open-ended VideoQA pairs grounded in 1,799 videos, spanning 137 tasks across 9 domains and five reasoning types covering preconditions, state evolution, counterfactuals, mistakes, and readiness. Benchmarking 13 leading MLLMs reveals a substantial answering-grounding gap: models produce plausible answers while failing to localize the supporting evidence (mIoU < 45%), exposing their reliance on linguistic priors rather than recognizing and reasoning about the object dynamics. As a step toward closing this gap, we further provide an object-centric supervised fine-tuning baseline with pseudo object-level supervision and spatial-temporal constraints. Models fine-tuned on ProcObject-10K not only improve on the benchmark itself, but also transfer effectively to other grounded VideoQA and embodied planning tasks. The dataset, annotations, and evaluation toolkit have been publicly released on our project page to support future research on object-centric procedural understanding: https://wenliangguo.github.io/ProcObject-10K-Page.
Jan 10, 2025

From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

Vision Language Models (VLMs) have achieved strong performance across a wide range of video understanding tasks. However, their viewpoint-invariant training limits their ability to infer egocentric properties, such as human-object interactions, from exocentric video observations. This limitation is particularly critical for applications such as Activities of Daily Living (ADL) monitoring, where understanding egocentric properties is essential but deploying wearable egocentric cameras is often impractical. We propose Ego2ExoVLM, a framework that enables VLMs to infer egocentric properties directly from exocentric videos by leveraging time-synchronized ego-exo video pairs during training. Our key insight is to treat the egocentric viewpoint as privileged supervision, providing rich interaction signals that are available only during training. Ego2ExoVLM consists of two complementary components: Ego2Exo Sequence Distillation, which transfers egocentric reasoning through a language-level sequence distillation objective, and Ego Adaptive Visual Tokens, which encourage the model to surface interaction-relevant cues within exocentric visual representations. To evaluate this capability, we introduce Ego-in-Exo Perception, a benchmark for assessing the understanding of egocentric properties from exocentric videos. We evaluate Ego2ExoVLM on 10 tasks spanning Ego-in-Exo Perception and existing ADL benchmarks, achieving state-of-the-art performance on the ADL-X benchmark suite and consistently outperforming strong baselines on our proposed benchmark. All code, models, and data will be released at https://github.com/dominickrei/EgoExo4ADL.