Geometry-Guided Camera Motion Understanding in VideoLLMs

Haoan Feng; Sri Harsha Musunuri; Guan-Ming Su

cs.CVMar 13, 2026

Geometry-Guided Camera Motion Understanding in VideoLLMs

Authors: Haoan Feng, Sri Harsha Musunuri, Guan-Ming Su

Organizations: University of Maryland, College Park · Dolby Laboratories Inc.

Abstract

Camera motion is a fundamental geometric signal that shapes visual perception and cinematic style, yet current video-capable vision-language models (VideoLLMs) rarely represent it explicitly and often fail on fine-grained motion primitives. We address this gap with a framework of $\textbf{benchmarking}$ , $\textbf{diagnosis}$ , and $\textbf{injection}$ . We curate $\textbf{CameraMotionDataset}$ , a large-scale synthetic dataset with explicit camera control, formulate camera motion as constraint-aware multi-label recognition, and construct a VQA benchmark-- $\textbf{CameraMotionVQA}$ . Across diverse off-the-shelf VideoLLMs, we observe substantial errors in recognizing camera motion primitives. Probing experiments on a Qwen2.5-VL vision encoder suggest that camera motion cues are weakly represented, especially in deeper ViT blocks, helping explain the observed failure modes. To bridge this gap without costly training or fine-tuning, we propose a lightweight, model-agnostic pipeline that extracts geometric camera cues from 3D foundation models (3DFMs), predicts constrained motion primitives with a temporal classifier, and injects them into downstream VideoLLM inference via structured prompting. Experiments demonstrate improved motion recognition and more camera-aware model responses, highlighting geometry-driven cue extraction and structured prompting as practical steps toward a camera-aware VideoLLM and VLA system. The dataset and benchmark is publicly available at https://hf.co/datasets/fengyee/camera-motion-dataset-and-benchmark.

Figures & tables

Figure 1: Overall pipeline. Camera cues are extracted from a frozen 3DFM (VGGT) and passed to a Transformer-based temporal classifier to predict camera-motion primitives, and per-second motions are injected as a structured prompt field for VideoLLMs without modifying VideoLLM weights. For clarity, the probing and distillation pipelines are shown separately in Fig. 3 and Fig. 4 .

Figure 2: Flow chart for dataset/benchmark construction. From MultiCamVideo [ 1 ] , video clips and camera extrinsics are preprocessed (split, resized, and normalized) and labeled with several motion constraints ( e.g . , incompatibility). A subset of shot segments is sampled based on motion primitives to balance classes, before storing as CameraMotionDataset and formulated into CameraMotionVQA records.

	Motion-level	Granularity	Temporal unit	QA protocol	Intended use	Camera params
CameraBench [ 3 ]	Primitive	Multi-label	Short clips	Caption, Yes/No QA	Benchmark + fine-tune	No
CineTechBench [ 7 ]	Composition	Coarse	Frames / short clips	Caption, MCQ	Benchmark	No
VidComposition [ 6 ]	Composition	Coarse	Compiled videos	MCQ	Benchmark	No
CineScale2 [ 4 ]	-	-	Frames	No	Training dataset	No
CameraMotionDataset	Primitive	Multi-label	1s within-shot	No	Training dataset	Yes
CameraMotionVQA	Primitive	Multi-label	1s within-shot	MCQ	Benchmark	No

Table 1: Positioning of our datasets against prior benchmarks and datasets. “Motion-level” indicates the granularity of camera supervision (primitive vs. composition of motions). Our CameraMotionDataset provides constrained primitive-level supervision with explicit synthetic camera parameters, while CameraMotionVQA converts the same 1s within-shot segments into a multiple-choice evaluation protocol. A full comparison with detailed annotation structure and intended usage is provided in the supplementary material.

Figure 3: Probing experiment schematic. Query tokens $Q_{t}$ gather camera motion-related information from the projected intermediate visual features of the frozen vision encoder. Camera motion logits are predicted from the temporal convolution output of the transferred vision tokens $Q_{t}^{\prime}$ .

Figure 4: VGGT–Q-Former schematic. Camera tokens and visual tokens are both bottlenecked by a projection layer. Query tokens gather camera motion-related information via interleaved local-/global-attention blocks and regress the projected camera tokens to distill the camera perception capability of VGGT. Branch annotated with Stage $i$ indicating different training optimization objectives and flows of tokens.

Figure 5: Off-the-shelf VideoLLM performance on CameraMotionVQA. Horizontal bars report overall multiple-choice accuracy. All models, labeled by their Hugging Face model names, use an identical frame input and VQA prompt template.

Method	Inst. Acc. $\uparrow$	Macro-F1 $\uparrow$	Weighted-F1 $\uparrow$
VGGT w. constraints	0.738	0.87	0.92
VGGT w/o. constraints	0.572	0.79	0.84
VGGT–Q-Former	0.638	0.83	0.87
Q-Former probing	0.450	0.69	0.74

Table 2: Multi-label camera-motion recognition results on the test split of CameraMotionDataset. Inst. Acc. : exact multi-label matching accuracy; Macro-F1 : class-averaged F1 score; Weighted-F1 : sample frequency-weighted F1 score.

Figure 6: Results of probing experiment. Intermediate features from the frozen vision tower at different Transformer block indices are probed by query tokens. Performance peaks at a shallow intermediate block and degrades in later layers, suggesting that camera-motion cues are not reliably preserved.

Figure 7: Structured motion header enables camera-aware and temporally grounded descriptions. Top: a 5-second clip sampled at 2 fps. Bottom: outputs from the same VideoLLM with different prompts. The baseline description correctly identifies subjects but produces ambiguous motion statements , without specifying direction or temporal structure . With structured motion cues, the model generates explicit camera usage , maintains subject focus, and introduces spatial-temporal reasoning such as continuity and coherence .

Pipeline	Params	Peak mem.	Throughput
	(M)	(MBs)	(samples/s)
VGGT classifier	9.47	23649.11	4.39
VGGT–Q-Former	9.15	9202.63	23.36
Q-Former probing	15.18	9232.23	25.12

Table 3: Efficiency comparison of camera-motion recognition pipelines. We compare trainable parameter count, peak GPU memory, and inference time throughput for: the full VGGT-based classifier; the distilled VGGT–Q-Former pipeline; and the Q-Former probing experiment. All measurements are conducted on an RTX A6000 with batch size 16 and input resolution

336\times 336

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Incompatibility matrix for constrained multi-label camera motion. Entry $\mathbf{M}_{ij}{=}1$ indicates that primitives $i$ and $j$ are mutually exclusive and cannot co-occur in the same 1-second segment. The axis structure produces block-wise exclusivity ( e.g . , left/right pairs), and static is incompatible with all non-static primitives. Row/column ordering matches the primitive index order used in our dataset and experiments.

Axis	Primitive	Operational Definition	Mutual exclusion
Yaw	pan left / pan right	Horizontal rotation about vertical axis	pan left $\leftrightarrow$ pan right
Pitch	tilt up / tilt down	Vertical rotation about horizontal axis	tilt up $\leftrightarrow$ tilt down
Roll	roll cw / roll ccw	In-plane rotation about optical axis	roll cw $\leftrightarrow$ roll ccw
Y	truck left / truck right	Lateral translation parallel to image plane	truck left $\leftrightarrow$ truck right
Z	crane up / crane down	Vertical translation of camera center	crane up $\leftrightarrow$ crane down
X	dolly in / dolly out	Translation along view axis	dolly in $\leftrightarrow$ dolly out

Appendix

Table 1: Atomic camera motion primitives. We list the

K{=}15

primitives used in our benchmark, grouped by geometric degree of freedom (rotations, translations, coupled motion, and static). Opposing directions within each axis are mutually exclusive, forming the basis of our incompatibility constraints, while primitives across different axes may co-occur under a constrained multi-label formulation.

Work	Primary focus	Temporal unit	Label type (constraints / QA)	Motion granularity	Intended use	Geometry access
CameraBench [ 3 ]	Camera motion primitives in diverse real-world videos; taxonomy + label-then-caption annotations.	Short video clips (avg. ~ 5–6s); shot-aligned / manually segmented.	Multi-label binary primitive tags ( ~ 50); plus captions and paired yes/no VQA skill probes.	Primitive-level; directional primitives (translation / rotation / zoom / tracking, etc.).	Mixed: benchmark suite (classification, retrieval, captioning, VQA) + small-scale training for motion-aware VLMs.	No : real videos (no ground-truth camera parameters).
CineTechBench [ 7 ]	Broad cinematographic techniques (scale, angle, composition, movement, lighting, color, focal length) with expert annotation.	Movie images + short movie clips/shots ( ~ 600 images, 120 clips).	Manual technique labels; question–answer pairs and annotated descriptions for understanding; separate camera-movement generation evaluation.	Coarse-to-mid: technique categories; camera movement is one dimension among 7.	Diagnostic benchmark (understanding + generation evaluation).	No : no explicit camera parameters provided.
VidComposition [ 6 ]	Composition understanding in compiled videos (cinematography, character, narrative, scene, making).	Compiled videos (avg. ~ 20 min) segmented into coherent sections.	Human-annotated multiple-choice QA (982 videos / 1706 questions) across 15 sub-tasks, including camera movement / angle / shot size perception.	Coarse-to-mid: composition attributes; camera motion appears as a perceptual attribute in QA.	Diagnostic benchmark for MLLMs on compiled-video composition.	No : no explicit camera parameters provided.
CineScale2 [ 4 ]	Cinematic camera features (camera angle + camera level) on movie frames/images.	Frames / images ( ~ 24.7k).	Single-label categorical annotations per attribute (angle: 5 classes; level: 6 classes).	N/A for motion (static camera setup attributes only).	Training-scale dataset for recognition of camera angle/level; film-style analysis.	No : no explicit camera parameters provided.
CameraMotionDataset	Camera motion primitive dataset for controlled supervision and motion-token distillation.	Within-shot 1s segments (fixed-length; shot-consistent).	Constrained multi-label primitive annotations with axis-wise incompatibility constraints.	Primitive-level; directional; axis-wise constraints ; supports compound motions within 1s.	Supervision dataset (training / distillation / controlled analysis).	Yes : synthetic camera parameters available (controlled rendering).
CameraMotionVQA	Camera-motion sensitivity benchmark for VideoLLM evaluation via structured prompting.	Same 1s within-shot segments mapped to QA format.	Standardized multiple-choice VQA protocol derived from ground-truth primitives; constraint-consistent answer space.	Primitive-level reasoning over directional motions (single and compound).	Diagnostic benchmark for motion-aware understanding and prompting.	Implicit (inherits synthetic geometry from CameraMotionDataset; not directly exposed to models).

Appendix

Table 2: Positioning of CameraMotionDataset and CameraMotionVQA against related benchmarks and cinematic datasets. We contrast task focus, temporal unit, label structure (including whether explicit constraints or QA protocols are provided), camera-motion granularity, intended use, and whether explicit camera parameters (geometry access) are available.

Figure 2: Label-set distribution before and after re-balancing. We cap each canonical primitive combination at 200 segments. The original distribution is highly skewed toward common directional motions, while the rebalanced dataset provides a more uniform coverage across primitive combinations.

Figure 3: Classifier capacity ablation. Accuracy versus classifier size (M parameters) for varying Transformer depth, head count, and hidden dimension. Overall performance saturates with small models; using 8 attention heads yields a modest but consistent gain over 4 heads.

Primitive	VGGT Classifier	VGGT– Q-Former	Q-Former Probing
arc ccw	0.88	0.83	0.68
arc cw	0.94	0.90	0.75
crane down	0.94	0.89	0.74
crane up	0.93	0.88	0.72
dolly in	0.94	0.90	0.78
dolly out	0.90	0.85	0.75

Appendix

Table 3: Per-primitive F1 scores. VGGT classifier achieves the best accuracy and consistently outperform the other two methods. A similar trend is visible between VGGT–Q-Former and Q-Former probing. However, the VGGT classifier requires running the 3D backbone, leading to higher computational and memory overhead. VGGT–Q-Former reuses frozen VideoLLM vision features and trades a modest accuracy drop for substantially higher throughput. Compared to probing, both VGGT-derived variants improve nearly all primitives, with the largest gains on ambiguous motions ( pan , tilt ) and the static class.

Figure 4: Cross-label confusion for the VGGT classifier. Rows denote ground-truth primitives and columns denote falsely predicted primitives (higher intensity indicates more frequent confusion). The label order matches Tab. 3 and the dataset primitive indices.

Figure 5: Clip2 sampled frames from CineTechBench [ 7 ] .

Figure 6: Clip3 sampled frames from CineTechBench [ 7 ] .

Baseline prompt	Structured prompt (filmmaker-style instruction)	Camera motion injected structured prompt
This video portrays a dramatic musical performance scene. It begins with a conductor, dressed in black, gesturing with precision and intensity, likely cueing the start of a musical piece. His facial expression is focused and assertive, signaling a moment of high tension. The camera quickly pans with motion blur , transitioning to a drummer mid-performance, energetically striking the drums while accompanied by a bassist. The motion blur appears again between frame changes, emphasizing swift camera movement or cuts to maintain rhythmic pacing. The conductor reappears in subsequent frames, continuing to direct with expressive hand movements, culminating in another quick blur before returning to the drummer, now deeply immersed in an intense and possibly climactic moment. The alternating shots and dynamic movement suggest a tightly edited sequence synchronized with musical tempo, evoking a sense of urgency and artistic fervor typical of a performance climax.	At the beginning, the video opens with a medium close-up of the conductor, dramatically lit with focused, warm stage lighting that casts soft shadows and isolates him against the dimly lit auditorium, emphasizing his stern facial expression and sharp, deliberate hand gestures. Then, a rapid whip pan to the right (evident from motion blur ) bridges the visual transition, dynamically redirecting the viewer’s focus to the drummer. The camera settles into a medium-wide shot of the drum set, capturing the drummer mid-performance under similarly warm, directional lighting that accentuates the shimmer of the cymbals and the energy of his movements. The framing includes part of the upright bassist, preserving ensemble context while maintaining the drummer as the focal point. Another whip pan follows, visually echoing the earlier motion and swiftly returning the viewer to the conductor. As the video cuts back , the conductor appears more animated in his movements, with tight framing and consistent lighting maintaining visual continuity. A final whip pan transitions back to the drummer, who now performs with increased intensity, reinforcing the escalating rhythm and emotional tension through repeated visual interplay. The sequence’s camera work –marked by fluid whip pans –mirrors the musical tempo, while the lighting and composition maintain dramatic cohesion, heightening the confrontation and synchrony between conductor and performer.	At the beginning, under warm, low-key stage lighting, a static medium closeup frames the conductor from stage right, isolating him against a softly lit auditorium background as he cues the ensemble with precise hand gestures. Then, a pan-left motion bridges the space, transitioning fluidly into a medium shot of the drummer, dynamically lit from above to emphasize the percussive energy and motion blur of his performance. Next, a pan-right swiftly returns the focus to the conductor, capturing his sharp directional cues in a near-symmetrical composition that mirrors the earlier frame, reinforcing continuity through motion. A static shot then holds on the conductor, now mid-gesture, in tighter framing that heightens dramatic tension while maintaining spatial coherence. Finally, a pan-left reintroduces the drummer in a more intensified moment–his kinetic energy visually amplified through rhythmic lighting and a centered composition that anchors the visual climax. The sequence skillfully employs lateral camera movement and repeated framing motifs to synchronize the visual rhythm with the musical performance, creating a cohesive and propulsive cinematic language.

Appendix

Table 4: Qualitative comparison of model-generated descriptions for Clip1 under different prompt templates. Camera motion-related phrases are highlighted in bold.

Baseline prompt	Structured prompt (filmmaker-style instruction)	Camera motion injected structured prompt
The clip shows a wounded man lying motionless on the forest floor among ferns and leaves, his face bruised and smeared with blood, suggesting he has been injured or unconscious. The camera gradually shifts its framing across the scene, revealing more of his body and the surrounding foliage as his head slowly tilts to the side. After remaining still for several moments, he begins to stir slightly, moving his head and hand as if regaining consciousness. Toward the end of the clip, another person enters from the left side, reaching toward him and leaning over his body, apparently checking on him or attempting to help him while the injured man weakly reacts.	At the beginning, the shot opens in a dim, cool-toned forest, lit by soft bluish ambient light. A tight overhead close-up frames a wounded man lying among ferns and damp leaves, his face near the center while foliage fills the surrounding space. As the sequence progresses, the camera slowly drifts laterally while rolling , shifting the man’s face from a frontal view to a slightly oblique angle. The framing gradually widens from a centered close-up to a looser, off-center composition, revealing more of his shoulder and the forest floor. The continuous roll with gentle tracking slides his head toward the lower part of the frame while foliage expands across the upper area, creating a drifting, hovering sensation. Toward the end, the camera dollies slightly outward while maintaining the roll , widening the overhead view as another person enters from the left and leans toward the injured man. The shot settles into a loose overhead two-shot , both figures partially visible among the ferns, with the slow rolling camera motion unifying the sequence into a calm but tense visual moment.	The shot opens in a dim, cool-toned forest with soft bluish lighting. A wounded man lies among ferns and damp leaves, framed in a tight overhead close-up that centers on his face and upper torso. The camera rolls slowly clockwise , keeping the close framing while rotating the horizon so the foliage appears to drift around him, creating a disorienting sense of instability. Midway through, the motion becomes more complex: the camera continues the clockwise roll while trucking left and craning up , gradually widening the view and revealing more of the forest floor. Soon after, the camera maintains the roll and begins a gentle dolly out , expanding the overhead composition. As the frame widens, another figure enters from the left, kneels beside the injured man, and reaches toward him. The shot ends in a wider overhead view, with both figures visible among the ferns as the slow rolling, outward-moving camera connects the sequence into a somber, atmospheric moment.

Appendix

Table 5: Qualitative comparison of model-generated descriptions for Clip2 under different prompt templates. Camera motion-related phrases are highlighted in bold.

Baseline prompt	Structured prompt (filmmaker-style instruction)	Camera motion injected structured prompt
A man wearing a long dark coat walks forward through a grand, modern throne room with floor-to-ceiling windows overlooking a futuristic city. Several guards in coordinated red uniforms stand symmetrically on both sides of the room, holding spears and watching as he approaches a large, curved ceremonial throne positioned at the center. Other armored attendants and officials stand nearby, observing quietly. The man slows as he reaches the throne, turns slightly, then lowers himself into the seat, gripping the armrests and settling into a commanding posture while the guards remain lined up behind him, reinforcing the formal and authoritative atmosphere of the scene.	At the beginning, a slightly canted medium-wide establishing shot inside a grand, glass-walled hall, where bright natural daylight floods in from the floor-to-ceiling windows behind the characters, producing strong backlighting and soft silhouettes. In the foreground, a man in a long dark coat walks toward a large circular throne-like structure positioned at the center of the frame, while symmetrical rows of red-armored guards stand rigidly in the background, forming a visual corridor that guides the viewer’s eye inward. As the shot progresses, the camera slowly tracks forward behind him, gradually straightening its angle and tightening the framing , turning the initial off-axis composition into a more centered, ceremonial perspective. The circular throne becomes a dominant compositional element, framing the character as he approaches; the guards remain evenly spaced on both sides, reinforcing the symmetry and depth of the mise-en-scène. Continuing the forward push-in , the camera subtly lowers and stabilizes into a frontal alignment as the man reaches the throne, placing him precisely on the central axis of the architecture and the guard formation. He then turns and lowers himself into the seat, the movement captured in a steady medium shot that emphasizes the ritualistic nature of the action. Finally, as he settles into the throne and grips the armrests, the camera completes its push-in to a centered, balanced composition: the character sits framed by the curved throne behind him, flanked symmetrically by the guards and the luminous cityscape beyond the windows, creating a powerful throne-room tableau under strong backlighting.	At the beginning, a slightly canted wide shot reveals a grand, glass-walled hall flooded with soft daylight from floor-to-ceiling windows, the bright skyline forming a luminous backdrop while a group of red-armored guards stands in symmetrical formation. A man in a dark coat walks toward a sculptural throne framed by two curved metallic blades. As the camera dollies in while rolling clockwise , the off-axis framing creates a subtle sense of motion and tension, gradually tightening the composition around the man’s back and the throne. The dolly-in with clockwise roll continues , bringing the throne into clearer alignment behind him while the guards remain evenly spaced on both sides, their red armor catching rim light from the windows. The lighting stays high-key and backlit, outlining figures with soft highlights while the foreground remains slightly shadowed, adding depth. As the camera keeps dollying in and rolling clockwise , the frame slowly levels and the throne becomes centered in the composition. The man turns and lowers himself toward the seat, the curved blades of the throne visually framing his body. Finally, the dolly-in with clockwise roll resolves into a centered medium shot , where he settles into the throne and grips the armrests, with the guards standing symmetrically behind him and the bright cityscape still glowing through the windows, completing the camera’s smooth approach and stabilizing the composition around his newly assumed position of authority.

Appendix

Table 6: Qualitative comparison of model-generated descriptions for Clip3 under different prompt templates. Camera motion-related phrases are highlighted in bold.

Figure 7: A visualization of the proposed data augmentations. For each augmentation, the left column shows the starting frames and the right column shows the augmented ending frames, because the augmentation is applied to the entire segment.

Panel of Figure 8 — Figure 8: The full prompt template for the LLM-as-judge evaluation protocol.

Prompt	CA	TC	RD	NM	LQ	Final
P1: Baseline	1	2	1	3	4	1.75
P2: Filmmaker	3	3	4	5	5	3.65
P3: + Motion labels	4	4	3	4	5	3.85

Appendix

Table 7: LLM-as-judge scores for three prompting strategies. Scores range from 1 (poor) to 5 (excellent). Injecting camera motion labels (P3) yields the largest gains in cinematographic accuracy and temporal continuity.

Camera Motion Understanding

See all 19 papers

Aug 11, 2026

Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation. Multimodal large language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a benchmark of 4,229 real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of 20 direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition. Project page: https://ddz16.github.io/cammotion.github.io/.

Dazhao Du, Shiyan Du, Jian Liu +8

The Hong Kong University of Science and Technology · Tencent

Temporal Video Grounding Camera Motion Understanding Multimodal Foundation Models

Ask PDF

Oct 12, 2026

When Left Is Right? Camera-Motion Understanding beyond Screen

Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/

Chong Cheng, Ran Zhang, Xinyuan Li +3

The Hong Kong University of Science and Technology (Guangzhou) · Tencent · LIX, Ecole Polytechnique, IP Paris

Camera Motion Understanding Video-Language Models Temporal Video Understanding Vision-Language Alignment VLM Evaluation

Ask PDF

Jul 3, 2026

Natural Language Camera Movement Understanding

Understanding camera movement in natural language is critical for training and evaluating video generation models, among other applications. However, we demonstrate that existing vision-language models (VLMs) fail this task in surprising ways, frequently confusing translation with rotation, left with right, and object movement with camera movement. To address these limitations, we establish natural language camera movement understanding as a standalone research task. We introduce a two-level cinematographic taxonomy and an extensive, atomic benchmark featuring both real and synthetic videos. Furthermore, we curate a large-scale, multi-source training set enhanced by targeted camera movement augmentation. Our fine-tuned VLM-8B outperforms Gemini 3.1 Pro by 10% and 11% on our benchmark's real and synthetic videos, respectively. Despite these gains, a significant gap remains relative to human performance, underscoring the need to promote and facilitate future research on natural language camera movement understanding.

Yuwen Tan, Joey Huang, Jin Huang +2

Boston University, Boston, MA 02215, USA · Pixocial Technology, Bellevue, WA 98004, USA

VLM Evaluation Video-Language Models Camera Motion Understanding

Ask PDF

Abstract

Camera Motion Understanding

Explore similar work

Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

When Left Is Right? Camera-Motion Understanding beyond Screen

Natural Language Camera Movement Understanding