Hybrid Cinematography: Previsualizing and Managing Hallucination Risk in Generative Video Reshooting

Nhan; Tran; Neal Wadhwa; Abe Davis; Stefan Stojanov

Figure 1. We present Hybrid Cinematography, a workflow connecting capture and generative post-production, built around one editable 3D shot plan that every stage reads (left). We demonstrate the workflow in a mobile augmented reality (AR) app. (a) Before or after capture, filmmakers plan the target move as keyframes by walking the scene, then set its timing. (b) In capture, the same plan drives the AR guidance. (c) In review, a previsualization (previs) checks the recorded take against that plan and marks what the model would have to invent, its hallucination risk . Every move between the captured move and the target move is a possible reshoot, asking the model to invent what the take does not support. Filmmakers navigate that range with a slider and see the risk of each move before anything is generated (right). They can then revise the move, capture more, re-time, or knowingly accept hallucination, while still on set. Video and examples: https://hybridcinematography.github.io/ Two-part figure. The left part, titled The Hybrid Cinematography Workflow, shows three stages connected in a loop around one 3D shot plan. In (a) Shot planning, people walk a scene and place camera keyframes in AR, joined by a dashed blue path. In (b) AR-guided capture, a person with a phone follows a trail of chevrons when far from a keyframe and matches its pose when near. In (c) Review and refine, a person holds a tablet that shows the previs and a slider. Two cards below the tablet compare the captured handheld path with the planned target path and show a slider that interpolates between them. A blue arrow labeled revise the same path and timing leads from review back to planning, and a dashed teal loop labeled guided additional capture leads from review back to capture. The right part, titled Previsualize hallucination risk before generating, enlarges the tablet screen. Three previs images sit at slider positions 0.00 (captured, lower risk), 0.67 (intermediate, balances capture and generation), and 1.00 (target, higher risk). Red hatching marks missing coverage and a blue-to-yellow heatmap marks resolution and performance deficits. Both grow from the captured end toward the target end. A card under each image shows the matching camera path and slider position. Four responses sit along the bottom: revise the move, capture more, re-time, and accept hallucination.

Workflow	Missing evidence found	Cost of a check	Responses
Generate and inspect	After generation	One generation	Revise the move, regenerate
Hybrid Cinematography, on set	Before generation	A proxy render	Revise, capture more, accept
Hybrid Cinematography, in editing	Before generation	A proxy render	Revise, accept

Table 2

Figure 2. Our hallucination risk previs estimates three deficits that vary with edits to the target camera move and indicate what downstream reshooting models must synthesize: (a) coverage for surfaces unseen from a compatible direction, (b) resolution for insufficient detail for the target framing, and (c) performance for missing performer evidence from a compatible viewpoint at the target instant. Three side-by-side constructions with recorded and target image pairs underneath. The scene contains three windows, a sofa, and a lamp. In coverage, parallel gray and blue cameras face the wall. The target translates to the right without rotating, and a newly revealed wall region is hatched red in both the construction and target image. In resolution, a target camera moves halfway toward the wall. A compact, unconnected distance scale in the white space beside the frustums marks the target depth z/2 halfway to the recorded depth z. A blue rectangle in the recorded image connects to the enlarged target crop, whose recorded samples are visibly softer while the recorded image remains sharp. In performance, a performer stands in front of the same wall with arms raised. A faded figure with lowered arms precedes it and a faded figure with a lifted knee follows it. Each time label sits below its pose, clear of the background. Gray and blue rays connect the selected instant to two cameras, whose images both show the raised-arm pose. The target image marks performer surfaces absent from the recorded view at that instant.

Figure 3. Comparing the previs, model conditioning, and generated output. Columns show the same take reshot as 0 ∘ , 15 ∘ , and 37 ∘ orbits. From top to bottom: the source frame, the previs showing hallucination risk (hatched red where no frame of the take saw the pixel, Viridis tint where resolution or performance falls short, brighter as the shortfall grows), Vista4D’s conditioning mask (white is conditioned) with our coverage estimate outlined in red, the generated output with synthesized regions circled, and pickup guidance. Wider orbits expose more missing content. The previs locates missing support before generation. It does not predict the quality of the synthesized content. Five rows by three columns. The columns are 0, 15, and 37 degree orbits of the same take. The rows show the captured frame, the previs with hatched red for missing coverage and a Viridis tint on the performer and room edges that grows with the orbit, the model's binary conditioning mask with a red outline where our coverage estimate overlaps its unconditioned pixels, the generated frames with circled synthesized regions, and a 3D view of the room cloud with the pickup camera frustum and aim ray.

Figure 4. Authoring a move in AR. Left : path. The filmmaker walks to each viewpoint and drops an AR keyframe. Each stretch between keyframes is a free move by default. Tapping a common shot-type preset, such as an orbit, crane, push, or dolly zoom, instead generates the keyframes for that move, and the system joins consecutive stretches into one continuous path. Right : timing. A speed curve over the same path defines camera velocity and holds. A double tap adds a control point, dragging it up or down speeds up or slows that stretch, and pulling it to the axis holds the camera still. Presets such as ease in-out apply a common profile to a stretch. Two landscape phone screenshots. Left: an AR view of a performer in an office with connected camera keyframes A, B, and C, and a panel offering Free, Orbit, Crane, Push, and Dolly zoom, with Push selected. Right: the same scene with a Timing panel showing a speed curve over keyframes A through C, a hold zone along the horizontal axis, a total duration of five seconds, and per-stretch easing options.

Figure 5. Coarse-to-fine AR guidance for capture, shown during a pickup. Left : coarse guidance. A trail of arrows on the floor leads the operator toward the saved viewpoint. Right : fine guidance. The saved camera’s frustum and a central crosshair help the operator match its position and framing. The same guidance supports rehearsal, the take, and pickup views saved during review. Two landscape phone screenshots of the Pickup interface in an office. Left: blue arrows form a trail across the floor toward a saved viewpoint. Right: a blue camera frustum, a yellow rectangular outline, and a yellow crosshair overlay the scene. Both show a banner asking the operator to point the camera at where the shot was authored to re-anchor the plan.

Figure 6. Review and Refine view. (A) The previs overlays the 3D proxy with hallucination risk: red hatching marks missing coverage, while a heatmap shows resolution and performance deficits. (B) controls playback. Toggling signals updates the overall risk map. (C) shows the captured and target trajectories and the interpolated moves selected with the compromise slider (D). Tapping a flagged region opens a popup with remedy strategies for (E) coverage or resolution gaps and (F) performance evidence gaps. Six labeled panels from the Review interface. The upper row has a large previs picture on the left with red hatched regions at the right edge and a transport bar labeled B overlaid near its bottom edge. On the right are a 3D risk map labeled C and a slider, risk strip, and signal controls labeled D. Below, E is a Coverage gap response card with Allow hallucination and Capture evidence buttons. F is a Performer evidence gap response card with Reduce move, Optimize timing, New take, and Allow hallucination buttons. Dashed white lines connect E to a red-hatched coverage region and F to the performer, meeting the bottom of each tap icon’s palm. Opaque white tap icons mark both selected regions. The coverage connector passes behind the timeline. Gold circles label the panels. The confirmation step is not shown.

Figure 7. The previs on ordinary video. Seven scenes recorded as RGB video without our application and reconstructed offline, with camera poses and depth from monocular estimation (Pi3X) and performer masks from video segmentation (SAM3), then reviewed in a browser viewer with the same map, keyframes, and slider as the app. Each row shows the captured path in black, an edited orbit as the target in dashed blue, the preview risk of the selected move at

\lambda=0.8

, and four successive frames of the previs for that move. Every orbit swings toward a side of the performer the take never filmed, so the performer wash rises toward yellow as the arc passes that side, while red hatching marks background the take never recorded and the same ramp on the background marks detail it sampled too coarsely. These are renders of what the take supports, not generated reshoots. Seven horizontal comic strips, one per scene: a studio performer, an office performer, hall cartwheel, lounge drink, a couple with a newspaper, conference punch, and a walking couple. Each strip starts with the actual camera-trajectory map and a captured-to-target slider, followed by four chronological reconstructed frames with warning overlays and timestamps. The map marker corresponds to the third frame. Red hatching marks unsupported proxy regions; blue-green through yellow tints indicate increasing resolution or performer-viewpoint warnings. Each row uses an edited target trajectory that moves around the performer to reveal unseen sides, at a fixed slider setting.

Figure 8. Study results, one color per participant. (A) Comfort with generative video reshooting before Part 1, after Part 1, and after Part 2. (B) Where each participant stopped on the Part 1 slider, between the captured move at 0 and the target move at 1. (C) Ratings of the on-set workflow after Part 2, one row per item. Item wording is abbreviated. The questions as asked are in Appendix E . Three panels. Panel A shows comfort at three points per participant. P1 3, 6, 6. P2 and P3 4, 5, 6. P4 5, 6, 6. P5 2, 4, 5. P6 5, 6, 6. P7 5, 7, 7. Medians 4, 6, 6. Panel B shows one row per participant on a track from 0, the captured move, to 1, the target move, with accepted stops at 0.67, 0.47, 0.49, 0.33, 0.19, 0.46, and 0.51 in participant order. Panel C shows nine rating items with one marker per participant on a 1 to 7 scale, in participant order. Easy to plan by walking and saving viewpoints 7, 7, 7, 6, 7, 6, 5. Easy to tune timing apart from the path 5, 5, 5, 7, 7, 7, 6. Guidance freed attention for the live scene 6, 6, 4, 6, 7, 6, 5. Clearly saw where and why AI must invent 7, 7, 7, 7, 7, 7, 5. Slider helped explore acceptable trade-offs 7, 6, 7, 5, 7, 6, 6. Warnings helped choose pickup, retake, or AI 6, 6, 7, 6, 7, 7, 5. Felt in control of creative decisions 6, 5, 6, 5, 5, 6, 5. On-set feedback worth the extra time 5, 6, 6, 5, 5, 7, 6. Confident wrapping, knowing footage support 3, 5, 7, 6, 6, 6, 6.

ID	Gender	Experience	Primary Roles	Production Domain	VFX and Generative AI
P1	M	9 years	Photographer, videographer	Social and brand content	Prior VFX collab with Disney, Pixar, ILM/Lucasfilm. Gen AI: Veo, Nano Banana
P2	M	6.5 years	Cinematographer (DP), grip, editor	Student and indie films	Rare VFX. Gen AI: open-source generative video models
P3	F	4 years	Video producer	Commercial and advertising	Occasional VFX, 3D modeling. Gen AI: Midjourney
P4	M	8 years	Cinematographer (DP), AC, gaffer, grip	Narrative short films	Never VFX. Gen AI: Midjourney, Topaz
P5	F	13 years	Cinematographer (DP), Assistant Camera (AC)	Narrative and indie films	No VFX. Gen AI: ChatGPT, Adobe AI tools
P6	M	7 years	Cinematographer (DP), Assistant Camera (AC)	Film, commercial shorts, documentaries, dance videos	VFX for extensions and cleanup. Gen AI: Seedance

Table 1. Film and video production backgrounds of the participants.

Figure 9. Part 1 interactive browser exercise. (a) Participants pin what they think a reshooting model invented in a generated view. (b) The reveal marks what the camera never recorded. (c) Participants navigate the range of possible reshoots with the previs, the map, and the slider. The green shaded region in the map marks the range of camera trajectories estimated to remain below the shot-support risk threshold. Three screenshots of the browser study. Left, a generated front view of a performer in an office with five numbered pins placed by a participant. Middle, the same view with a red hatch over never-recorded surfaces and a yellow mask over the face. Right, the study scene two thirds of the way through the take with the slider two thirds toward the planned move: the reconstructed take with the coverage hatch and the performer washed yellow, with a popup reading that the camera never filmed his face here and the AI must invent his facial features, beside a top-down map with keyframes and the compromise slider.

Figure 10. Inputs, shared representations, and outputs of the workflow. (a) Video, depth, camera poses and intrinsics, and performer masks provide the recorded evidence; path and timing specify the editable shot plan. (b) The plan is placed in a point-cloud proxy of the scene and performer. (c) This shared scene and plan support AR capture guidance and shot-support review before generation. (d) After review, the source video, proxy, and selected camera trajectory provide the inputs for downstream conditioning and offline reshooting. The proxy is rendered along the selected trajectory to prepare model-specific conditioning views and masks. All illustrations are schematic. A compact data-flow diagram with four columns. Inputs show RGB, depth, and performer-mask views, camera poses and intrinsics, and two square panels for path and timing. These feed a dotted room and performer with the planned camera path. A shared connector branches to boxes for AR capture guidance and previs with shot-support review, and to a downstream reshooting model in a separate post-production column. The source RGB video also connects directly to the model, without repeating the inputs. The model outputs a reshot video.

Item	Statement topic	P1	P2	P3	P4	P5	P6	P7	Median (min–max)
Planning and capture
1.1	Easy to plan by walking, saving viewpoints	7	7	7	6	7	6	5	7 (5–7)
1.2	Easy to tune timing apart from the path	5	5	5	7	7	7	6	6 (5–7)
2.1	Guidance freed attention for the live scene	6	6	4	6	7	6	5	6 (4–7)
Review and decisions
3.1	Clearly saw where and why AI must invent	7	7	7	7	7	7	5	7 (5–7)

Hybrid Cinematography: Previsualizing and Managing Hallucination Risk in Generative Video Reshooting

Abstract

Figures & tables

Appendix

Explore similar work

Addressing Image Authenticity When Cameras Use Generative AI

HALLELUAI: A Hallucination-Aware AI System for Ultra-Realistic Image-to-Video Generation at Scale

ShotPlan: Cinematic Video Generation with Learnable Planning Token