cs.CVMar 13, 2026
SaveGeometry-Guided Camera Motion Understanding in VideoLLMs
Organizations: University of Maryland, College Park · Dolby Laboratories Inc.
Abstract
Camera motion is a fundamental geometric signal that shapes visual perception and cinematic style, yet current video-capable vision-language models (VideoLLMs) rarely represent it explicitly and often fail on fine-grained motion primitives. We address this gap with a framework of , , and . We curate , a large-scale synthetic dataset with explicit camera control, formulate camera motion as constraint-aware multi-label recognition, and construct a VQA benchmark--. Across diverse off-the-shelf VideoLLMs, we observe substantial errors in recognizing camera motion primitives. Probing experiments on a Qwen2.5-VL vision encoder suggest that camera motion cues are weakly represented, especially in deeper ViT blocks, helping explain the observed failure modes. To bridge this gap without costly training or fine-tuning, we propose a lightweight, model-agnostic pipeline that extracts geometric camera cues from 3D foundation models (3DFMs), predicts constrained motion primitives with a temporal classifier, and injects them into downstream VideoLLM inference via structured prompting. Experiments demonstrate improved motion recognition and more camera-aware model responses, highlighting geometry-driven cue extraction and structured prompting as practical steps toward a camera-aware VideoLLM and VLA system. The dataset and benchmark is publicly available at https://hf.co/datasets/fengyee/camera-motion-dataset-and-benchmark.
Figures & tables
Figure 1: Overall pipeline. Camera cues are extracted from a frozen 3DFM (VGGT) and passed to a Transformer-based temporal classifier to predict camera-motion primitives, and per-second motions are injected as a structured prompt field for VideoLLMs without modifying VideoLLM weights. For clarity, the probing and distillation pipelines are shown separately in Fig. 3 and Fig. 4 .
Figure 2: Flow chart for dataset/benchmark construction. From MultiCamVideo [ 1 ] , video clips and camera extrinsics are preprocessed (split, resized, and normalized) and labeled with several motion constraints ( e.g . , incompatibility). A subset of shot segments is sampled based on motion primitives to balance classes, before storing as CameraMotionDataset and formulated into CameraMotionVQA records.
| Motion-level | Granularity | Temporal unit | QA protocol | Intended use | Camera params | |
| CameraBench [ 3 ] | Primitive | Multi-label | Short clips | Caption, Yes/No QA | Benchmark + fine-tune | No |
| CineTechBench [ 7 ] | Composition | Coarse | Frames / short clips | Caption, MCQ | Benchmark | No |
| VidComposition [ 6 ] | Composition | Coarse | Compiled videos | MCQ | Benchmark | No |
| CineScale2 [ 4 ] | - | - | Frames | No | Training dataset | No |
| CameraMotionDataset | Primitive | Multi-label | 1s within-shot | No | Training dataset | Yes |
| CameraMotionVQA | Primitive | Multi-label | 1s within-shot | MCQ | Benchmark | No |
Table 1: Positioning of our datasets against prior benchmarks and datasets. “Motion-level” indicates the granularity of camera supervision (primitive vs. composition of motions). Our CameraMotionDataset provides constrained primitive-level supervision with explicit synthetic camera parameters, while CameraMotionVQA converts the same 1s within-shot segments into a multiple-choice evaluation protocol. A full comparison with detailed annotation structure and intended usage is provided in the supplementary material.
Figure 3: Probing experiment schematic. Query tokens gather camera motion-related information from the projected intermediate visual features of the frozen vision encoder. Camera motion logits are predicted from the temporal convolution output of the transferred vision tokens .
Figure 4: VGGT–Q-Former schematic. Camera tokens and visual tokens are both bottlenecked by a projection layer. Query tokens gather camera motion-related information via interleaved local-/global-attention blocks and regress the projected camera tokens to distill the camera perception capability of VGGT. Branch annotated with Stage indicating different training optimization objectives and flows of tokens.
Figure 5: Off-the-shelf VideoLLM performance on CameraMotionVQA. Horizontal bars report overall multiple-choice accuracy. All models, labeled by their Hugging Face model names, use an identical frame input and VQA prompt template.
| Method | Inst. Acc. | Macro-F1 | Weighted-F1 |
|---|---|---|---|
| VGGT w. constraints | 0.738 | 0.87 | 0.92 |
| VGGT w/o. constraints | 0.572 | 0.79 | 0.84 |
| VGGT–Q-Former | 0.638 | 0.83 | 0.87 |
| Q-Former probing | 0.450 | 0.69 | 0.74 |
Table 2: Multi-label camera-motion recognition results on the test split of CameraMotionDataset. Inst. Acc. : exact multi-label matching accuracy; Macro-F1 : class-averaged F1 score; Weighted-F1 : sample frequency-weighted F1 score.
Figure 6: Results of probing experiment. Intermediate features from the frozen vision tower at different Transformer block indices are probed by query tokens. Performance peaks at a shallow intermediate block and degrades in later layers, suggesting that camera-motion cues are not reliably preserved.
Figure 7: Structured motion header enables camera-aware and temporally grounded descriptions. Top: a 5-second clip sampled at 2 fps. Bottom: outputs from the same VideoLLM with different prompts. The baseline description correctly identifies subjects but produces ambiguous motion statements , without specifying direction or temporal structure . With structured motion cues, the model generates explicit camera usage , maintains subject focus, and introduces spatial-temporal reasoning such as continuity and coherence .
| Pipeline | Params | Peak mem. | Throughput |
|---|---|---|---|
| (M) | (MBs) | (samples/s) | |
| VGGT classifier | 9.47 | 23649.11 | 4.39 |
| VGGT–Q-Former | 9.15 | 9202.63 | 23.36 |
| Q-Former probing | 15.18 | 9232.23 | 25.12 |
Table 3: Efficiency comparison of camera-motion recognition pipelines. We compare trainable parameter count, peak GPU memory, and inference time throughput for: the full VGGT-based classifier; the distilled VGGT–Q-Former pipeline; and the Q-Former probing experiment. All measurements are conducted on an RTX A6000 with batch size 16 and input resolution .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Incompatibility matrix for constrained multi-label camera motion. Entry indicates that primitives and are mutually exclusive and cannot co-occur in the same 1-second segment. The axis structure produces block-wise exclusivity ( e.g . , left/right pairs), and static is incompatible with all non-static primitives. Row/column ordering matches the primitive index order used in our dataset and experiments.
| Axis | Primitive | Operational Definition | Mutual exclusion |
|---|---|---|---|
| Yaw | pan left / pan right | Horizontal rotation about vertical axis | pan left pan right |
| Pitch | tilt up / tilt down | Vertical rotation about horizontal axis | tilt up tilt down |
| Roll | roll cw / roll ccw | In-plane rotation about optical axis | roll cw roll ccw |
| Y | truck left / truck right | Lateral translation parallel to image plane | truck left truck right |
| Z | crane up / crane down | Vertical translation of camera center | crane up crane down |
| X | dolly in / dolly out | Translation along view axis | dolly in dolly out |
Table 1: Atomic camera motion primitives. We list the primitives used in our benchmark, grouped by geometric degree of freedom (rotations, translations, coupled motion, and static). Opposing directions within each axis are mutually exclusive, forming the basis of our incompatibility constraints, while primitives across different axes may co-occur under a constrained multi-label formulation.
| Work | Primary focus | Temporal unit | Label type (constraints / QA) | Motion granularity | Intended use | Geometry access |
| CameraBench [ 3 ] | Camera motion primitives in diverse real-world videos; taxonomy + label-then-caption annotations. | Short video clips (avg. ~ 5–6s); shot-aligned / manually segmented. | Multi-label binary primitive tags ( ~ 50); plus captions and paired yes/no VQA skill probes. | Primitive-level; directional primitives (translation / rotation / zoom / tracking, etc.). | Mixed: benchmark suite (classification, retrieval, captioning, VQA) + small-scale training for motion-aware VLMs. | No : real videos (no ground-truth camera parameters). |
| CineTechBench [ 7 ] | Broad cinematographic techniques (scale, angle, composition, movement, lighting, color, focal length) with expert annotation. | Movie images + short movie clips/shots ( ~ 600 images, 120 clips). | Manual technique labels; question–answer pairs and annotated descriptions for understanding; separate camera-movement generation evaluation. | Coarse-to-mid: technique categories; camera movement is one dimension among 7. | Diagnostic benchmark (understanding + generation evaluation). | No : no explicit camera parameters provided. |
| VidComposition [ 6 ] | Composition understanding in compiled videos (cinematography, character, narrative, scene, making). | Compiled videos (avg. ~ 20 min) segmented into coherent sections. | Human-annotated multiple-choice QA (982 videos / 1706 questions) across 15 sub-tasks, including camera movement / angle / shot size perception. | Coarse-to-mid: composition attributes; camera motion appears as a perceptual attribute in QA. | Diagnostic benchmark for MLLMs on compiled-video composition. | No : no explicit camera parameters provided. |
| CineScale2 [ 4 ] | Cinematic camera features (camera angle + camera level) on movie frames/images. | Frames / images ( ~ 24.7k). | Single-label categorical annotations per attribute (angle: 5 classes; level: 6 classes). | N/A for motion (static camera setup attributes only). | Training-scale dataset for recognition of camera angle/level; film-style analysis. | No : no explicit camera parameters provided. |
| CameraMotionDataset | Camera motion primitive dataset for controlled supervision and motion-token distillation. | Within-shot 1s segments (fixed-length; shot-consistent). | Constrained multi-label primitive annotations with axis-wise incompatibility constraints. | Primitive-level; directional; axis-wise constraints ; supports compound motions within 1s. | Supervision dataset (training / distillation / controlled analysis). | Yes : synthetic camera parameters available (controlled rendering). |
| CameraMotionVQA | Camera-motion sensitivity benchmark for VideoLLM evaluation via structured prompting. | Same 1s within-shot segments mapped to QA format. | Standardized multiple-choice VQA protocol derived from ground-truth primitives; constraint-consistent answer space. | Primitive-level reasoning over directional motions (single and compound). | Diagnostic benchmark for motion-aware understanding and prompting. | Implicit (inherits synthetic geometry from CameraMotionDataset; not directly exposed to models). |
Table 2: Positioning of CameraMotionDataset and CameraMotionVQA against related benchmarks and cinematic datasets. We contrast task focus, temporal unit, label structure (including whether explicit constraints or QA protocols are provided), camera-motion granularity, intended use, and whether explicit camera parameters (geometry access) are available.
Figure 2: Label-set distribution before and after re-balancing. We cap each canonical primitive combination at 200 segments. The original distribution is highly skewed toward common directional motions, while the rebalanced dataset provides a more uniform coverage across primitive combinations.
Figure 3: Classifier capacity ablation. Accuracy versus classifier size (M parameters) for varying Transformer depth, head count, and hidden dimension. Overall performance saturates with small models; using 8 attention heads yields a modest but consistent gain over 4 heads.
| Primitive | VGGT Classifier | VGGT– Q-Former | Q-Former Probing |
|---|---|---|---|
| arc ccw | 0.88 | 0.83 | 0.68 |
| arc cw | 0.94 | 0.90 | 0.75 |
| crane down | 0.94 | 0.89 | 0.74 |
| crane up | 0.93 | 0.88 | 0.72 |
| dolly in | 0.94 | 0.90 | 0.78 |
| dolly out | 0.90 | 0.85 | 0.75 |
Table 3: Per-primitive F1 scores. VGGT classifier achieves the best accuracy and consistently outperform the other two methods. A similar trend is visible between VGGT–Q-Former and Q-Former probing. However, the VGGT classifier requires running the 3D backbone, leading to higher computational and memory overhead. VGGT–Q-Former reuses frozen VideoLLM vision features and trades a modest accuracy drop for substantially higher throughput. Compared to probing, both VGGT-derived variants improve nearly all primitives, with the largest gains on ambiguous motions ( pan , tilt ) and the static class.
Figure 4: Cross-label confusion for the VGGT classifier. Rows denote ground-truth primitives and columns denote falsely predicted primitives (higher intensity indicates more frequent confusion). The label order matches Tab. 3 and the dataset primitive indices.
Figure 5: Clip2 sampled frames from CineTechBench [ 7 ] .
Figure 6: Clip3 sampled frames from CineTechBench [ 7 ] .
| Baseline prompt | Structured prompt (filmmaker-style instruction) | Camera motion injected structured prompt |
| This video portrays a dramatic musical performance scene. It begins with a conductor, dressed in black, gesturing with precision and intensity, likely cueing the start of a musical piece. His facial expression is focused and assertive, signaling a moment of high tension. The camera quickly pans with motion blur , transitioning to a drummer mid-performance, energetically striking the drums while accompanied by a bassist. The motion blur appears again between frame changes, emphasizing swift camera movement or cuts to maintain rhythmic pacing. The conductor reappears in subsequent frames, continuing to direct with expressive hand movements, culminating in another quick blur before returning to the drummer, now deeply immersed in an intense and possibly climactic moment. The alternating shots and dynamic movement suggest a tightly edited sequence synchronized with musical tempo, evoking a sense of urgency and artistic fervor typical of a performance climax. | At the beginning, the video opens with a medium close-up of the conductor, dramatically lit with focused, warm stage lighting that casts soft shadows and isolates him against the dimly lit auditorium, emphasizing his stern facial expression and sharp, deliberate hand gestures. Then, a rapid whip pan to the right (evident from motion blur ) bridges the visual transition, dynamically redirecting the viewer’s focus to the drummer. The camera settles into a medium-wide shot of the drum set, capturing the drummer mid-performance under similarly warm, directional lighting that accentuates the shimmer of the cymbals and the energy of his movements. The framing includes part of the upright bassist, preserving ensemble context while maintaining the drummer as the focal point. Another whip pan follows, visually echoing the earlier motion and swiftly returning the viewer to the conductor. As the video cuts back , the conductor appears more animated in his movements, with tight framing and consistent lighting maintaining visual continuity. A final whip pan transitions back to the drummer, who now performs with increased intensity, reinforcing the escalating rhythm and emotional tension through repeated visual interplay. The sequence’s camera work –marked by fluid whip pans –mirrors the musical tempo, while the lighting and composition maintain dramatic cohesion, heightening the confrontation and synchrony between conductor and performer. | At the beginning, under warm, low-key stage lighting, a static medium closeup frames the conductor from stage right, isolating him against a softly lit auditorium background as he cues the ensemble with precise hand gestures. Then, a pan-left motion bridges the space, transitioning fluidly into a medium shot of the drummer, dynamically lit from above to emphasize the percussive energy and motion blur of his performance. Next, a pan-right swiftly returns the focus to the conductor, capturing his sharp directional cues in a near-symmetrical composition that mirrors the earlier frame, reinforcing continuity through motion. A static shot then holds on the conductor, now mid-gesture, in tighter framing that heightens dramatic tension while maintaining spatial coherence. Finally, a pan-left reintroduces the drummer in a more intensified moment–his kinetic energy visually amplified through rhythmic lighting and a centered composition that anchors the visual climax. The sequence skillfully employs lateral camera movement and repeated framing motifs to synchronize the visual rhythm with the musical performance, creating a cohesive and propulsive cinematic language. |
Table 4: Qualitative comparison of model-generated descriptions for Clip1 under different prompt templates. Camera motion-related phrases are highlighted in bold.
| Baseline prompt | Structured prompt (filmmaker-style instruction) | Camera motion injected structured prompt |
| The clip shows a wounded man lying motionless on the forest floor among ferns and leaves, his face bruised and smeared with blood, suggesting he has been injured or unconscious. The camera gradually shifts its framing across the scene, revealing more of his body and the surrounding foliage as his head slowly tilts to the side. After remaining still for several moments, he begins to stir slightly, moving his head and hand as if regaining consciousness. Toward the end of the clip, another person enters from the left side, reaching toward him and leaning over his body, apparently checking on him or attempting to help him while the injured man weakly reacts. | At the beginning, the shot opens in a dim, cool-toned forest, lit by soft bluish ambient light. A tight overhead close-up frames a wounded man lying among ferns and damp leaves, his face near the center while foliage fills the surrounding space. As the sequence progresses, the camera slowly drifts laterally while rolling , shifting the man’s face from a frontal view to a slightly oblique angle. The framing gradually widens from a centered close-up to a looser, off-center composition, revealing more of his shoulder and the forest floor. The continuous roll with gentle tracking slides his head toward the lower part of the frame while foliage expands across the upper area, creating a drifting, hovering sensation. Toward the end, the camera dollies slightly outward while maintaining the roll , widening the overhead view as another person enters from the left and leans toward the injured man. The shot settles into a loose overhead two-shot , both figures partially visible among the ferns, with the slow rolling camera motion unifying the sequence into a calm but tense visual moment. | The shot opens in a dim, cool-toned forest with soft bluish lighting. A wounded man lies among ferns and damp leaves, framed in a tight overhead close-up that centers on his face and upper torso. The camera rolls slowly clockwise , keeping the close framing while rotating the horizon so the foliage appears to drift around him, creating a disorienting sense of instability. Midway through, the motion becomes more complex: the camera continues the clockwise roll while trucking left and craning up , gradually widening the view and revealing more of the forest floor. Soon after, the camera maintains the roll and begins a gentle dolly out , expanding the overhead composition. As the frame widens, another figure enters from the left, kneels beside the injured man, and reaches toward him. The shot ends in a wider overhead view, with both figures visible among the ferns as the slow rolling, outward-moving camera connects the sequence into a somber, atmospheric moment. |
Table 5: Qualitative comparison of model-generated descriptions for Clip2 under different prompt templates. Camera motion-related phrases are highlighted in bold.
| Baseline prompt | Structured prompt (filmmaker-style instruction) | Camera motion injected structured prompt |
| A man wearing a long dark coat walks forward through a grand, modern throne room with floor-to-ceiling windows overlooking a futuristic city. Several guards in coordinated red uniforms stand symmetrically on both sides of the room, holding spears and watching as he approaches a large, curved ceremonial throne positioned at the center. Other armored attendants and officials stand nearby, observing quietly. The man slows as he reaches the throne, turns slightly, then lowers himself into the seat, gripping the armrests and settling into a commanding posture while the guards remain lined up behind him, reinforcing the formal and authoritative atmosphere of the scene. | At the beginning, a slightly canted medium-wide establishing shot inside a grand, glass-walled hall, where bright natural daylight floods in from the floor-to-ceiling windows behind the characters, producing strong backlighting and soft silhouettes. In the foreground, a man in a long dark coat walks toward a large circular throne-like structure positioned at the center of the frame, while symmetrical rows of red-armored guards stand rigidly in the background, forming a visual corridor that guides the viewer’s eye inward. As the shot progresses, the camera slowly tracks forward behind him, gradually straightening its angle and tightening the framing , turning the initial off-axis composition into a more centered, ceremonial perspective. The circular throne becomes a dominant compositional element, framing the character as he approaches; the guards remain evenly spaced on both sides, reinforcing the symmetry and depth of the mise-en-scène. Continuing the forward push-in , the camera subtly lowers and stabilizes into a frontal alignment as the man reaches the throne, placing him precisely on the central axis of the architecture and the guard formation. He then turns and lowers himself into the seat, the movement captured in a steady medium shot that emphasizes the ritualistic nature of the action. Finally, as he settles into the throne and grips the armrests, the camera completes its push-in to a centered, balanced composition: the character sits framed by the curved throne behind him, flanked symmetrically by the guards and the luminous cityscape beyond the windows, creating a powerful throne-room tableau under strong backlighting. | At the beginning, a slightly canted wide shot reveals a grand, glass-walled hall flooded with soft daylight from floor-to-ceiling windows, the bright skyline forming a luminous backdrop while a group of red-armored guards stands in symmetrical formation. A man in a dark coat walks toward a sculptural throne framed by two curved metallic blades. As the camera dollies in while rolling clockwise , the off-axis framing creates a subtle sense of motion and tension, gradually tightening the composition around the man’s back and the throne. The dolly-in with clockwise roll continues , bringing the throne into clearer alignment behind him while the guards remain evenly spaced on both sides, their red armor catching rim light from the windows. The lighting stays high-key and backlit, outlining figures with soft highlights while the foreground remains slightly shadowed, adding depth. As the camera keeps dollying in and rolling clockwise , the frame slowly levels and the throne becomes centered in the composition. The man turns and lowers himself toward the seat, the curved blades of the throne visually framing his body. Finally, the dolly-in with clockwise roll resolves into a centered medium shot , where he settles into the throne and grips the armrests, with the guards standing symmetrically behind him and the bright cityscape still glowing through the windows, completing the camera’s smooth approach and stabilizing the composition around his newly assumed position of authority. |
Table 6: Qualitative comparison of model-generated descriptions for Clip3 under different prompt templates. Camera motion-related phrases are highlighted in bold.
Figure 7: A visualization of the proposed data augmentations. For each augmentation, the left column shows the starting frames and the right column shows the augmented ending frames, because the augmentation is applied to the entire segment.
Figure 8: The full prompt template for the LLM-as-judge evaluation protocol.
| Prompt | CA | TC | RD | NM | LQ | Final |
|---|---|---|---|---|---|---|
| P1: Baseline | 1 | 2 | 1 | 3 | 4 | 1.75 |
| P2: Filmmaker | 3 | 3 | 4 | 5 | 5 | 3.65 |
| P3: + Motion labels | 4 | 4 | 3 | 4 | 5 | 3.85 |
Table 7: LLM-as-judge scores for three prompting strategies. Scores range from 1 (poor) to 5 (excellent). Injecting camera motion labels (P3) yields the largest gains in cinematographic accuracy and temporal continuity.