Video Understanding: From Geometry and Semantics to Unified Models

Zhaochong An; Zirui Li; Mingqiao Ye; Feng Qiao; Jiaang Li; Zongwei Wu; Vishal Thengane; Chengzu Li

cs.CVMar 18, 2026

Video Understanding: From Geometry and Semantics to Unified Models

Authors: Zhaochong An, Zirui Li, Mingqiao Ye, Feng Qiao, Jiaang Li, Zongwei Wu, Vishal Thengane, Chengzu Li,

Organizations: Department of Computer Science, University of Copenhagen, Copenhagen 2100, Denmark. · College of Computer Science, Nankai University, Tianjin 300350, China. · School of Computer and Communication Sciences, EPFL, Lausanne 1015, Switzerland. · Department of Computer Science & Engineering, Washington University in St. Louis, St. Louis MO 63130, USA. · Computer Vision Lab, University of Würzburg, Würzburg 97070, Germany. · Computer Science Research Centre, University of Surrey, Guildford GU2 7XH, UK. · School of Electrical, Computer and Telecommunications Engineering , University of Wollongong, Wollongong 2500, Australia. · Language Technology Lab, University of Cambridge, Cambridge CB2 1TN, UK. · School of Artificial Intelligence, Beijing Institute of Technology, Beijing 100081, China. · Institute for Computer Science, INSAIT, Sofia 1784, Bulgaria.

Abstract

Video understanding aims to enable models to perceive, reason about, and interact with the dynamic visual world. In contrast to image understanding, video understanding inherently requires modeling temporal dynamics and evolving visual context, placing stronger demands on spatiotemporal reasoning and making it a foundational problem in computer vision. In this survey, we present a structured overview of video understanding by organizing the literature into three complementary perspectives: low-level video geometry understanding, high-level semantic understanding, and unified video understanding models. We further highlight a broader shift from isolated, task-specific pipelines toward unified modeling paradigms that can be adapted to diverse downstream objectives, enabling a more systematic view of recent progress. By consolidating these perspectives, this survey provides a coherent map of the evolving video understanding landscape, summarizes key modeling trends and design principles, and outlines open challenges toward building robust, scalable, and unified video foundation models.

Figures & tables

Figure 1: Classification of video understanding by level. This survey organizes video understanding methods into three categories: (1) low-level geometry understanding, (2) high-level semantic understanding, and (3) unified video understanding. The right diagram illustrates the conceptual relationship among the three levels.

Figure 2: Comparison of low-level video geometry understanding tasks : video depth estimation ( left ), camera pose estimation ( middle ), and optical flow/point tracking ( right ). While all three tasks take video frames as input, they recover different geometric quantities, namely scene depth, camera motion, and temporal correspondences across frames.

Method	Category	Abs Rel $\downarrow$	$\delta\uparrow$
DepthFormer li2023depthformer	Feed-forward	0.058	0.967
ManyDepth watson2021temporal	Feed-forward	0.069	0.930
MAMo yasarla2023mamo	Feed-forward	0.049	0.949
NVDS NVDS	Inference align.	0.253	0.588
ChronoDepth chronodepth	Diffusion-based	0.167	0.759
Depth-Anything depth_anything_v1	Feed-forward	0.142	0.803

Table 1: Video depth estimation on KITTI geiger2013vision . Methods are grouped according to the taxonomy in Section 2.1 : inference-time alignment, feed-forward prediction, and diffusion-based approaches. We report Absolute Relative Error (Abs Rel,

\downarrow

) and the accuracy threshold

\delta<1.25

(

\uparrow

). Arrows indicate whether higher or lower values are better. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.

Method	Category	$t_{\mathrm{err}}\downarrow$	$r_{\mathrm{err}}\downarrow$
DenseVLAD torii201524	Correspondence	0.26	13.11
Geo. PN kendall2017geometric	Regression (APR)	0.23	8.12
PoseNet kendall2015posenet	Regression (APR)	0.21	7.74
DFNet chen2022dfnet	Regression (APR)	0.02	0.79
LENS moreau2022lens	Regression (APR)	0.08	3.00
ESSNet zhou2020learn	Correspondence	0.22	11.82

Table 2: Camera pose estimation on 7 Scenes shotton2013scene . Methods are grouped according to the taxonomy in Section 2.2 into correspondence-and-solver pipelines and pose regression approaches, including absolute (APR) and relative (RPR) formulations. We report median translation error

t_{\mathrm{err}}

(metres,

\downarrow

) and median rotation error

r_{\mathrm{err}}

(degrees,

\downarrow

) to reflect camera relocalisation accuracy. Lower values indicate better accuracy. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.

Method	Category	AJ $\uparrow$	$\delta_{\mathrm{avg}}\uparrow$	OA $\uparrow$
MFT mft	Dense flow-based	47.3	66.8	77.8
TAPIR doersch2023tapir	Offline TAP	56.2	70.0	86.5
SpatialTracker xiao2024spatialtracker	3D-consistent	61.1	76.3	89.5
TAPTRv2 li2024taptrv2	Offline TAP	63.5	75.9	91.4
CoTracker3 karaev2024cotracker3	Offline TAP	64.5	76.7	89.7
Track-On aydemir2025track	Online TAP	65.0	78.0	90.8

Table 3: Optical flow and point tracking on DAVIS pont20172017 . Methods are categorized according to Section 2.3 into dense flow-based tracking, offline Tracking Any Point (TAP), and online TAP. Tracking quality is evaluated using Average Jaccard (AJ,

\uparrow

), average displacement accuracy

\delta_{\mathrm{avg}}

(

\uparrow

), and overall accuracy (OA,

\uparrow

). Higher values indicate better performance. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.

Figure 3: Comparison between different joint feed-forward geometry models. (a) Pairwise processing methods weinzaepfel2022croco ; wang2024dust3r predict view-consistent representations ( e.g. , point maps) from image pairs, typically operating on independent frame pairs and relying on later aggregation or optimization to get multi-view geometry information. (b) Multi-view-input, multi-output methods wang2025vggt ; cut3r jointly reason over a set of frames in a single forward pass, using global attention or memory to directly predict multiple geometric primitives with cross-view consistency.

Method	Type	Video Depth (VD)				Camera Pose		3D Reconstruction
		KITTI		Sintel		7 Scenes	RSE10K	Co3D v2		NRGBD
		Abs Rel $\downarrow$	$\delta\uparrow$	Abs Rel $\downarrow$	$\delta\uparrow$	A30 $\uparrow$	A30 $\uparrow$	Acc. $\downarrow$	Comp. $\downarrow$	Acc. $\downarrow$	Comp. $\downarrow$
DUST3R wang2024dust3r	Pair & MV	0.143	0.814	0.653	44.9	76.7	67.7	0.077	0.067	0.019	0.018
MASt3R leroy2024grounding	Pair & MV	0.115	0.848	0.641	43.9	81.8	76.4	0.081	0.069	0.033	0.028
MONST3R zhang2024monst3r	Dynamic	0.107	0.884	0.358	54.8	–	–	0.185	0.167	0.114	0.110
Spann3R wang2024spann3r	Dynamic	0.198	0.837	0.622	42.6	60.8	70.4	0.226	0.112	0.323	0.285

Table 4: Comparison of feed-forward geometry models across video depth, camera pose, and 3D reconstruction tasks. VD denotes Video Depth Estimation . For VD benchmarks ( KITTI and Sintel ), performance is evaluated using Absolute Relative Error (Abs Rel,

\downarrow

) and the depth accuracy metric

\delta

(

\uparrow

), where

\delta

denotes the percentage of pixels satisfying

\max(\hat{d}/d,d/\hat{d})<1.25

. Camera pose estimation is evaluated on 7 Scenes and RealEstate10K using A30 (

\uparrow

), defined as the area under the error curve up to

30^{\circ}

for translation and rotation errors. 3D reconstruction performance is reported on Co3D v2 and NRGBD using reconstruction accuracy (Acc.,

\downarrow

) and completeness (Comp.,

\downarrow

). Arrows indicate whether higher or lower values are better. “Pair & MV” denotes methods based on pairwise or multi-view paradigms, while “Dynamic” refers to approaches designed for dynamic scenes and online deployment. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.

Figure 4: Overview of video segmentation. We categorize video segmentation methods into three paradigms based on how semantic categories are specified. (a) Video class-aware segmentation assumes a fixed, predefined label set and includes video semantic, instance, and panoptic segmentation. (b) Video open-vocabulary segmentation extends class-aware settings by leveraging language embeddings to segment both seen and unseen categories. (c) Video class-agnostic segmentation removes semantic labels altogether, instead segmenting and tracking objects using visual or concept-level prompts such as clicks, boxes, or masks.

Figure 5: Overview of video object tracking. (a) Siamese tracking performs target localization by matching a template extracted from the target object against a search region in the current frame. (b) Sequence-level tracking extends this formulation by modeling temporal dependencies across multiple consecutive frames, thereby improving tracking continuity and robustness. (c) Multimodal tracking further incorporates complementary modalities, such as visual, textual, or auxiliary sensory cues, to enhance target representation and achieve more reliable tracking in challenging scenarios.

Method	Modality	DepthTrack yan2021depthtrack			LasHeR li2021lasher		RGBT234 li2019rgb		VisEvent wang2023visevent
Method	Modality	F-score $\uparrow$	Recall $\uparrow$	Precision $\uparrow$	Precision $\uparrow$	AUC $\uparrow$	MPR $\uparrow$	MSR $\uparrow$	Precision $\uparrow$	AUC $\uparrow$
ViPT zhu2023visual	Miss	44.4	40.5	46.6	40.1	34.0	52.4	39.4	57.2	43.2
ViPT zhu2023visual	Full	59.4 (+15.0)	59.6 (+19.1)	59.2 (+12.6)	65.1 (+25.0)	52.5 (+18.5)	83.5 (+31.1)	61.7 (+22.3)	75.8 (+18.6)	59.2 (+16.0)
SDSTrack hou2024sdstrack	Miss	46.7	42.0	52.7	52.5	43.1	67.0	48.8	62.6	46.9
SDSTrack hou2024sdstrack	Full	61.4 (+14.7)	60.9 (+18.9)	61.9 (+9.2)	66.5 (+14.0)	53.1 (+10.0)	84.8 (+17.8)	62.5 (+13.7)	76.7 (+14.1)	59.7 (+12.8)
STTrack hu2025exploiting	Miss	49.9	48.8	51.0	54.5	44.9	73.8	54.2	65.5	49.7

Table 5: Performance comparison on four multimodal tracking benchmarks under Full (complete modalities) and Miss (temporally missing modalities) settings, as introduced in tan2025you . Missing-setting results are reported on the corresponding * miss benchmark variants including DepthTrack miss , LasHeR miss , RGBT234 miss , and VisEvent miss . Performance gaps between Full and Miss settings are highlighted in green ; smaller gaps indicate stronger robustness to modality absence. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.

Method	Backbone	Initialization / Pre-training	Input size	Speed (FPS)	Hardware
ViPT zhu2023visual	ViT-B/16 dosovitskiy2020image	OSTrack initialization	Template 128 Search 256	24.78	A6000
SDSTrack hou2024sdstrack	ViT-B/16 dosovitskiy2020image	OSTrack initialization	Template 128 Search 256	20.86	A6000
STTrack hu2025exploiting	ViT-B/16 dosovitskiy2020image	SOT foundation model	Template 128 Search 256	—	—
SUTrack chen2024sutrack	HiViT-B zhang2022hivit	Fast-iTPN initialization (HiViT) + CLIP-L text encoder	Template 112/192 Search 224/384	12.0	RTX 2080Ti
FlexTrack tan2025you	Video encoder + HMoE fusion	Large-scale pretrained encoder	—	—	—

Table 6: Key configurations and inference efficiency of representative multimodal trackers. Reported FPS values are taken from the original papers and should be interpreted with care, as backbones, input resolutions, pre-training settings, and test hardware may differ across methods.

Method	Paradigm	Architecture	Core Mechanism	Objective / Signal
TGN chen2018temporally	Supervised	3D-CNN tran2015learning + GloVe pennington2014glove	Sequential Aggregation (LSTM/GRU)	Cross-Entropy Classification
ABLR yuan2019to	Supervised	3D-CNN + GloVe	Multimodal Cross-Attention	Smooth L1 Regression
TGA mithun2019weakly	Weakly Sup.	CLIP radford2021learning + GloVe	Text-Guided Attention	Reconstruction Loss
WSDEC duan2018weakly	Weakly Sup.	Sentences localizer + generator	Dual-Task (Captioning + Localization)	Cycle-Consistency
TFVTG zheng2024training	Zero-Shot	BLIP-2 li2023blip / CLIP	Sub-Event Decomposition	Logic-based Filtering
Moment-GPT xu2025zero	Zero-Shot	Video-ChatGPT Maaz2023VideoChatGPT	Generative Scoring	Generation Confidence

Table 7: Comparative Analysis of Representative Video Temporal Grounding Methods. The shift from TGN chen2018temporally to ABLR yuan2019to represents a shift from classification logic to regression logic while TGA mithun2019weakly and WSDEC duan2018weakly represent the supervision shift towards the weakly supervision manner. Furthermore, later approaches increasingly intervene in the model’s internal reasoning process: TFVTG zheng2024training incorporates logic scripts, Moment-GPT xu2025zero leverages ensemble reasoning, and Time-R1 wang2025time adopts reward-based optimization. Overall, the field is moving toward deeper and more intrinsic control over model reasoning, rather than relying primarily on input manipulation or output-level heuristics. Results are taken from original papers, and experimental settings may differ; strict comparisons require reference to the original protocols.

Figure 6: An illustration of video temporal grounding for the action. The 10-second clip captures the transition of the fritter from being held by a fork to being successfully released into the cooking vessel. The highlighted frames demonstrate the model’s ability to precisely localize the start and end points of this specific interaction.

Figure 7: Video Question Answering. The VideoQA task requires a model to jointly reason over visual and linguistic inputs. Given a video $V$ composed of temporal frames and a question $Q$ , the system extracts video features and language representations, which are then fused within a multimodal model to generate a grounded natural language answer $A$ .

Category	Representative Works	Key Focus & Technical Contributions
Evolution of Benchmarks
Descriptive	MSVD-QA xu2017video , MSRVTT-QA xu2017video	Factoid Retrieval: Recognition of atomic actions and objects in short clips limited by static bias.
Temporal & Causal	TGIF-QA Jang_2017_CVPR , NExT-QA xiao2021next , ActivityNet-QA yu2019activitynet	Relational Logic: Tasks for repetition counting, state transitions, and causal interaction graphs like why and how.
Long-Form Narrative	EgoSchema mangalam2023egoschema , CinePile rawal2024cinepile , Video-MME fu2025video , LongVideoBench wu2024longvideobench , MLVU zhou2024mlvu	Needle-in-a-Haystack: Multi-hop reasoning over hour-scale contexts penalizing single-frame shortcuts via subset sensitivity.
Spatial & Active	VSI-SUPER yang2025cambrian , StreamingCoT hu2025streamingcot	World Modeling: Tests long-horizon object permanence known as Spatial Supersensing and dynamic Chain-of-Thought.
Architectural Paradigms

Table 8: Summary of VideoQA Benchmarks and Architectural Paradigms. The field has evolved from short-term descriptive tasks to long-form, predictive, and spatial reasoning challenges.

Figure 8: Different framework designs of unified video understanding and generation models : (a) Assembled systems rely on an LLM to orchestrate external video expert models. (b) Autoregressive (AR) unified models map text, image, and video into a single token stream, processing them through a unified transformer with a discrete next-token prediction objective. (c) Hybrid unified models utilize a shared transformer backbone but incorporate continuous generation targets ( e.g. , diffusion or flow matching) via VAE decoders to better match the demands of video synthesis.

Model	Venue	Year	Backbone	Unified	Paradigm
HuggingGPT shen2023hugginggpt	NeurIPS	2023	GPT-3.5	Assembled	Tool
NExT-GPT wu2024next	ICML	2024	Vicuna chiang2023vicuna	Assembled	AR*
Video-LaVIT jin2024video	ACL	2024	LLaMA-2 touvron2023llama	Native	AR
Emu3 wang2024emu3	arXiv	2024	LLaMA-2	Native	AR
VILA-U wu2024vila	ICLR	2025	LLaMA-2	Native	AR
BAGEL deng2025emerging	arXiv	2025	Qwen-2.5 yang2025qwen3	Native	AR+Flow

Table 9: Unified video understanding and generation models. “Tool” denotes LLM-orchestrated external expert models (not end-to-end). “*” indicates external generation modules (not end-to-end). Paradigms: autoregressive (AR), diffusion (Diff), and flow matching (Flow).

Video Foundation Models

See all 16 papers

Watch, Remember, Reason: Human-View Video Understanding with MLLMs
Jun 5, 2026Jiahao Meng, Yue Tan, Qi Xu +12Streaming Video Understanding Video-Language Models
School of Intelligence Science and Technology, Peking University · Wuhan University · Shanghai Jiao Tong University +7
Video-Oasis: Rethinking Evaluation of Video Understanding
Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5VLM Evaluation Temporal Video Understanding
Sejong University, South Korea · NAVER Cloud, South Korea
What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models
Sep 1, 2026Sharon S. Musa, Fereshteh Forghani, Harrish Thasarathan +3Video Representation Learning Vision Foundation Models
Lassonde School of Engineering, York University, Toronto, Canada · Vector Institute, Toronto, Canada · FAIR, Meta Superintelligence Labs +2

Jun 5, 2026J/K move · Enter open · S save

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Jiahao Meng, Yue Tan, Qi Xu, Kuan Gao+11

School of Intelligence Science and Technology, Peking University · Wuhan University · Shanghai Jiao Tong University+7

Streaming Video Understanding Video-Language Models Long-Video Understanding Video Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.

Abstract

Video Foundation Models

Explore similar work

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Video-Oasis: Rethinking Evaluation of Video Understanding

What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models