Video Understanding: From Geometry and Semantics to Unified Models
Authors: Zhaochong An, Zirui Li, Mingqiao Ye, Feng Qiao, Jiaang Li, Zongwei Wu, Vishal Thengane, Chengzu Li, Lei Li, Luc Van Gool, Guolei Sun, Serge Belongie
Organizations: Department of Computer Science, University of Copenhagen, Copenhagen 2100, Denmark. · College of Computer Science, Nankai University, Tianjin 300350, China. · School of Computer and Communication Sciences, EPFL, Lausanne 1015, Switzerland. · Department of Computer Science & Engineering, Washington University in St. Louis, St. Louis MO 63130, USA. · Computer Vision Lab, University of Würzburg, Würzburg 97070, Germany. · Computer Science Research Centre, University of Surrey, Guildford GU2 7XH, UK. · School of Electrical, Computer and Telecommunications Engineering , University of Wollongong, Wollongong 2500, Australia. · Language Technology Lab, University of Cambridge, Cambridge CB2 1TN, UK. · School of Artificial Intelligence, Beijing Institute of Technology, Beijing 100081, China. · Institute for Computer Science, INSAIT, Sofia 1784, Bulgaria.
Video understanding aims to enable models to perceive, reason about, and interact with the dynamic visual world. In contrast to image understanding, video understanding inherently requires modeling temporal dynamics and evolving visual context, placing stronger demands on spatiotemporal reasoning and making it a foundational problem in computer vision. In this survey, we present a structured overview of video understanding by organizing the literature into three complementary perspectives: low-level video geometry understanding, high-level semantic understanding, and unified video understanding models. We further highlight a broader shift from isolated, task-specific pipelines toward unified modeling paradigms that can be adapted to diverse downstream objectives, enabling a more systematic view of recent progress. By consolidating these perspectives, this survey provides a coherent map of the evolving video understanding landscape, summarizes key modeling trends and design principles, and outlines open challenges toward building robust, scalable, and unified video foundation models.
Figures & tables
Figure 1: Classification of video understanding by level. This survey organizes video understanding methods into three categories: (1) low-level geometry understanding, (2) high-level semantic understanding, and (3) unified video understanding. The right diagram illustrates the conceptual relationship among the three levels.
Figure 2: Comparison of low-level video geometry understanding tasks : video depth estimation ( left ), camera pose estimation ( middle ), and optical flow/point tracking ( right ). While all three tasks take video frames as input, they recover different geometric quantities, namely scene depth, camera motion, and temporal correspondences across frames.
Method
Category
Abs Rel ↓
δ↑
DepthFormer li2023depthformer
Feed-forward
0.058
0.967
ManyDepth watson2021temporal
Feed-forward
0.069
0.930
MAMo yasarla2023mamo
Feed-forward
0.049
0.949
NVDS NVDS
Inference align.
0.253
0.588
ChronoDepth chronodepth
Diffusion-based
0.167
0.759
Depth-Anything depth_anything_v1
Feed-forward
0.142
0.803
Table 1: Video depth estimation on KITTI geiger2013vision . Methods are grouped according to the taxonomy in Section 2.1 : inference-time alignment, feed-forward prediction, and diffusion-based approaches. We report Absolute Relative Error (Abs Rel, ↓ ) and the accuracy threshold δ<1.25 ( ↑ ). Arrows indicate whether higher or lower values are better. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.
Method
Category
terr↓
rerr↓
DenseVLAD torii201524
Correspondence
0.26
13.11
Geo. PN kendall2017geometric
Regression (APR)
0.23
8.12
PoseNet kendall2015posenet
Regression (APR)
0.21
7.74
DFNet chen2022dfnet
Regression (APR)
0.02
0.79
LENS moreau2022lens
Regression (APR)
0.08
3.00
ESSNet zhou2020learn
Correspondence
0.22
11.82
Table 2: Camera pose estimation on 7 Scenes shotton2013scene . Methods are grouped according to the taxonomy in Section 2.2 into correspondence-and-solver pipelines and pose regression approaches, including absolute (APR) and relative (RPR) formulations. We report median translation error terr (metres, ↓ ) and median rotation error rerr (degrees, ↓ ) to reflect camera relocalisation accuracy. Lower values indicate better accuracy. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.
Method
Category
AJ ↑
δavg↑
OA ↑
MFT mft
Dense flow-based
47.3
66.8
77.8
TAPIR doersch2023tapir
Offline TAP
56.2
70.0
86.5
SpatialTracker xiao2024spatialtracker
3D-consistent
61.1
76.3
89.5
TAPTRv2 li2024taptrv2
Offline TAP
63.5
75.9
91.4
CoTracker3 karaev2024cotracker3
Offline TAP
64.5
76.7
89.7
Track-On aydemir2025track
Online TAP
65.0
78.0
90.8
Table 3: Optical flow and point tracking on DAVIS pont20172017 . Methods are categorized according to Section 2.3 into dense flow-based tracking, offline Tracking Any Point (TAP), and online TAP. Tracking quality is evaluated using Average Jaccard (AJ, ↑ ), average displacement accuracy δavg ( ↑ ), and overall accuracy (OA, ↑ ). Higher values indicate better performance. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.
Figure 3: Comparison between different joint feed-forward geometry models. (a) Pairwise processing methods weinzaepfel2022croco ; wang2024dust3r predict view-consistent representations ( e.g. , point maps) from image pairs, typically operating on independent frame pairs and relying on later aggregation or optimization to get multi-view geometry information. (b) Multi-view-input, multi-output methods wang2025vggt ; cut3r jointly reason over a set of frames in a single forward pass, using global attention or memory to directly predict multiple geometric primitives with cross-view consistency.
Method
Type
Video Depth (VD)
Camera Pose
3D Reconstruction
KITTI
Sintel
7 Scenes
RSE10K
Co3D v2
NRGBD
Abs Rel ↓
δ↑
Abs Rel ↓
δ↑
A30 ↑
A30 ↑
Acc. ↓
Comp. ↓
Acc. ↓
Comp. ↓
DUST3R wang2024dust3r
Pair & MV
0.143
0.814
0.653
44.9
76.7
67.7
0.077
0.067
0.019
0.018
MASt3R leroy2024grounding
Pair & MV
0.115
0.848
0.641
43.9
81.8
76.4
0.081
0.069
0.033
0.028
MONST3R zhang2024monst3r
Dynamic
0.107
0.884
0.358
54.8
–
–
0.185
0.167
0.114
0.110
Spann3R wang2024spann3r
Dynamic
0.198
0.837
0.622
42.6
60.8
70.4
0.226
0.112
0.323
0.285
Table 4: Comparison of feed-forward geometry models across video depth, camera pose, and 3D reconstruction tasks. VD denotes Video Depth Estimation . For VD benchmarks ( KITTI and Sintel ), performance is evaluated using Absolute Relative Error (Abs Rel, ↓ ) and the depth accuracy metric δ ( ↑ ), where δ denotes the percentage of pixels satisfying max(d^/d,d/d^)<1.25 . Camera pose estimation is evaluated on 7 Scenes and RealEstate10K using A30 ( ↑ ), defined as the area under the error curve up to 30∘ for translation and rotation errors. 3D reconstruction performance is reported on Co3D v2 and NRGBD using reconstruction accuracy (Acc., ↓ ) and completeness (Comp., ↓ ). Arrows indicate whether higher or lower values are better. “Pair & MV” denotes methods based on pairwise or multi-view paradigms, while “Dynamic” refers to approaches designed for dynamic scenes and online deployment. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.
Figure 4: Overview of video segmentation. We categorize video segmentation methods into three paradigms based on how semantic categories are specified. (a) Video class-aware segmentation assumes a fixed, predefined label set and includes video semantic, instance, and panoptic segmentation. (b) Video open-vocabulary segmentation extends class-aware settings by leveraging language embeddings to segment both seen and unseen categories. (c) Video class-agnostic segmentation removes semantic labels altogether, instead segmenting and tracking objects using visual or concept-level prompts such as clicks, boxes, or masks.
Figure 5: Overview of video object tracking. (a) Siamese tracking performs target localization by matching a template extracted from the target object against a search region in the current frame. (b) Sequence-level tracking extends this formulation by modeling temporal dependencies across multiple consecutive frames, thereby improving tracking continuity and robustness. (c) Multimodal tracking further incorporates complementary modalities, such as visual, textual, or auxiliary sensory cues, to enhance target representation and achieve more reliable tracking in challenging scenarios.
Method
Modality
DepthTrack yan2021depthtrack
LasHeR li2021lasher
RGBT234 li2019rgb
VisEvent wang2023visevent
F-score ↑
Recall ↑
Precision ↑
Precision ↑
AUC ↑
MPR ↑
MSR ↑
Precision ↑
AUC ↑
ViPT zhu2023visual
Miss
44.4
40.5
46.6
40.1
34.0
52.4
39.4
57.2
43.2
Full
59.4 (+15.0)
59.6 (+19.1)
59.2 (+12.6)
65.1 (+25.0)
52.5 (+18.5)
83.5 (+31.1)
61.7 (+22.3)
75.8 (+18.6)
59.2 (+16.0)
SDSTrack hou2024sdstrack
Miss
46.7
42.0
52.7
52.5
43.1
67.0
48.8
62.6
46.9
Full
61.4 (+14.7)
60.9 (+18.9)
61.9 (+9.2)
66.5 (+14.0)
53.1 (+10.0)
84.8 (+17.8)
62.5 (+13.7)
76.7 (+14.1)
59.7 (+12.8)
STTrack hu2025exploiting
Miss
49.9
48.8
51.0
54.5
44.9
73.8
54.2
65.5
49.7
Table 5: Performance comparison on four multimodal tracking benchmarks under Full (complete modalities) and Miss (temporally missing modalities) settings, as introduced in tan2025you . Missing-setting results are reported on the corresponding * miss benchmark variants including DepthTrack miss , LasHeR miss , RGBT234 miss , and VisEvent miss . Performance gaps between Full and Miss settings are highlighted in green ; smaller gaps indicate stronger robustness to modality absence. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.
Method
Backbone
Initialization / Pre-training
Input size
Speed (FPS)
Hardware
ViPT zhu2023visual
ViT-B/16 dosovitskiy2020image
OSTrack initialization
Template 128 Search 256
24.78
A6000
SDSTrack hou2024sdstrack
ViT-B/16 dosovitskiy2020image
OSTrack initialization
Template 128 Search 256
20.86
A6000
STTrack hu2025exploiting
ViT-B/16 dosovitskiy2020image
SOT foundation model
Template 128 Search 256
—
—
SUTrack chen2024sutrack
HiViT-B zhang2022hivit
Fast-iTPN initialization (HiViT) + CLIP-L text encoder
Template 112/192 Search 224/384
12.0
RTX 2080Ti
FlexTrack tan2025you
Video encoder + HMoE fusion
Large-scale pretrained encoder
—
—
—
Table 6: Key configurations and inference efficiency of representative multimodal trackers. Reported FPS values are taken from the original papers and should be interpreted with care, as backbones, input resolutions, pre-training settings, and test hardware may differ across methods.
Table 7: Comparative Analysis of Representative Video Temporal Grounding Methods. The shift from TGN chen2018temporally to ABLR yuan2019to represents a shift from classification logic to regression logic while TGA mithun2019weakly and WSDEC duan2018weakly represent the supervision shift towards the weakly supervision manner. Furthermore, later approaches increasingly intervene in the model’s internal reasoning process: TFVTG zheng2024training incorporates logic scripts, Moment-GPT xu2025zero leverages ensemble reasoning, and Time-R1 wang2025time adopts reward-based optimization. Overall, the field is moving toward deeper and more intrinsic control over model reasoning, rather than relying primarily on input manipulation or output-level heuristics. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.
Figure 6: An illustration of video temporal grounding for the action. The 10-second clip captures the transition of the fritter from being held by a fork to being successfully released into the cooking vessel. The highlighted frames demonstrate the model’s ability to precisely localize the start and end points of this specific interaction.
Figure 7: Video Question Answering. The VideoQA task requires a model to jointly reason over visual and linguistic inputs. Given a video V composed of temporal frames and a question Q , the system extracts video features and language representations, which are then fused within a multimodal model to generate a grounded natural language answer A .
Category
Representative Works
Key Focus & Technical Contributions
Evolution of Benchmarks
Descriptive
MSVD-QA xu2017video , MSRVTT-QA xu2017video
Factoid Retrieval: Recognition of atomic actions and objects in short clips limited by static bias.
World Modeling: Tests long-horizon object permanence known as Spatial Supersensing and dynamic Chain-of-Thought.
Architectural Paradigms
Table 8: Summary of VideoQA Benchmarks and Architectural Paradigms. The field has evolved from short-term descriptive tasks to long-form, predictive, and spatial reasoning challenges.
Figure 8: Different framework designs of unified video understanding and generation models : (a) Assembled systems rely on an LLM to orchestrate external video expert models. (b) Autoregressive (AR) unified models map text, image, and video into a single token stream, processing them through a unified transformer with a discrete next-token prediction objective. (c) Hybrid unified models utilize a shared transformer backbone but incorporate continuous generation targets ( e.g. , diffusion or flow matching) via VAE decoders to better match the demands of video synthesis.
Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.
Jiahao Meng, Yue Tan, Qi Xu +12
School of Intelligence Science and Technology, Peking University · Wuhan University · Shanghai Jiao Tong University +7
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.
Geuntaek Lim, Sungjune Park, Jaeyun Lee +5
Sejong University, South Korea · NAVER Cloud, South Korea
Self-supervised video foundation models learn rich spatiotemporal representations, yet it remains unclear what visual concepts these representations encode, where they emerge across transformer layers, and how they are geometrically organized. In this work, we tackle these three questions through a systematic layer-wise analysis of V-JEPA 2 and VideoMAE-v2. We leverage lightweight probes trained to discover three temporally grounded properties: (i) camera motion understanding, (ii) intuitive physics, and (iii) anomaly detection. Both models encode camera motion, with best results (>90 ROC AUC) emerging at 60-70% of network depth, and achieve moderate anomaly detection performance (>60 ROC AUC), but remain near chance on intuitive-physics tasks, suggesting a limited encoding of deeper physical reasoning. Beyond classification, we find that temporal features from individual videos form smooth low-dimensional trajectories in representation space, suggesting that camera motion is not only linearly decodable but also geometrically organized. Based on these results, we apply geometry-aware spline-based steering in the model's latent representations to interpolate camera motion, yielding steered videos with smoother trajectories and more coherent temporal progression than linear interpolation.
Sharon S. Musa, Fereshteh Forghani, Harrish Thasarathan +3
Lassonde School of Engineering, York University, Toronto, Canada · Vector Institute, Toronto, Canada · FAIR, Meta Superintelligence Labs +2