We present Gen Through the Lens, a camera-controllable image-to-video generation framework that synthesizes videos from a given reference image while enabling realistic and physically consistent camera operations. Unlike previous models that treat camera motion as abstract conditioning or rely on a single coordinate system, our approach explicitly decomposes and fuses control signals across three coordinate spaces: the camera coordinate (modeled by intrinsic parameters), the world coordinate (represented by spatial geometry hints), and the object coordinate (described by 3D bounding boxes of target subjects). Through cross-coordinate conditioning fusion, these heterogeneous cues are unified to guide the generative process, enabling precise control of camera motion, focus, and framing while preserving temporal and spatial coherence. Extensive experiments demonstrate that our method produces controllable and visually consistent results across diverse camera operations, outperforming existing approaches in both realism and editability.
Figures & tables
Figure 1. Overview of Gen Through the Lens , the first image-to-video generation framework that jointly enables controllable camera motion, object motion, and depth-of-field effects. Starting from a single input image, our model supports cinematic camera trajectories, zoom and focus transitions, and stable object dynamics in a physically consistent manner. The lower left shows a modeling software environment for editing camera trajectories, object motion, and depth-of-field information. The second column shows the three complementary conditioning signals, and the right side shows the generated video results, with the model’s key controllable capabilities summarized below.
Figure 2. Overview of the data processing pipeline. Recovered metric-scale camera parameters and human-centered 3D geometry are converted into geometry-aware conditions for camera motion, object motion, and depth-of-field control.
Figure 3. Overview of the proposed framework. The model jointly conditions video generation on camera motion, object motion, and depth-of-field signals across camera, world, and object coordinate spaces.
Method
Cam Trans. Err. ↓
Cam Rot. Err. (°) ↓
VGGT
41.25
20.34
Pi3
40.36
17.21
MapAnything
12.39
17.43
Ours
11.74
17.11
Table 1. Metric-scale camera pose accuracy on the MultiCamVideo Dataset with ground-truth camera annotations. Methods that estimate camera poses only up to scale (VGGT and Pi3) exhibit large errors on sequences with significant camera motion. Our method recovers metrically consistent camera parameters and achieves lower errors, validating the effectiveness of the proposed pipeline.
Method
FID ↓
FVD ↓
Qwen VQA Score ↑
Cam Trans. Err. ↓
Cam Rot. Err. (°) ↓
mIoU ↑
BBox Center Err. (%) ↓
BBox Rot. Err. (°) ↓
Traj. Err. (%) ↓
PAC
70.17
172.51
0.56
18.15
8.77
0.52
6.37
41.83
8.68
RC-I2V
55.12
98.76
0.70
9.86
4.63
0.45∗
7.42∗
47.34∗
7.29
GEN3C
64.24
114.33
0.62
11.38
5.02
0.47∗
8.24∗
46.22∗
8.39
GTL 14B
53.71
86.54
0.81
8.59
4.89
0.61
4.40
21.73
6.37
Table 2. Quantitative comparison with state-of-the-art methods. Lower is better ( ↓ ) for error-based metrics and latent space distance, while higher is better ( ↑ ) for overlap and semantic metrics. Screen-space errors are reported as percentages of the image diagonal. Metrics marked with ∗ should be interpreted with caution, as RealCam-I2V and GEN3C do not support explicit control of object motion or target object positioning, which may bias object-related metrics toward worse values.
Figure 4. Qualitative comparison of camera controllability with state-of-the-art methods. Gen3C and RealCam-I2V use identical camera motion extracted from the original video, with the first frame as input. CamCloneMaster and Seedance 2.0 use the original video as the camera reference. Our Camera condition encodes recovered extrinsics and intrinsics. For clarity, text prompts are omitted in this figure and provided in the Appendix Sec. E .
Figure 5. Qualitative comparison of object motion control. We compare our method with Perception-as-Control under aligned camera conditions and comparable object motion controls, focusing on the controllability and stability of foreground object motion. For clarity, text prompts are omitted in this figure and provided in the Appendix Sec. E .
Figure 6. Disentanglement between camera and object motion. Keeping the object fixed in screen space under camera motion requires the object to move together with the camera in 3D. The prompts used here avoid specifying object motion and are provided in Appendix Sec. E .
Table 9
Figure 7. Visualization of depth-of-field conditions and generated results. Our representation enables controllable bokeh strength, time-varying focus shifts, and stable depth-of-field under camera and object motion. Note that focus cues are encoded through subtle pixel-level variations, which may not be visually salient. The corresponding text prompts are provided in Appendix Sec. E .
Figure 8. Hitchcock zoom effect synthesized by our model. In this example, the camera moves backward by 3 meters while the zoom is adjusted accordingly to approximately preserve the projected size of the subject in the image, as indicated by the object motion condition video. The corresponding text prompts are provided in Appendix Sec. E .
Figure 9. Additional qualitative results corresponding to the experiments. Text prompts are omitted in this figure and provided in the Appendix Sec. E .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Camera Extraction Method
Cam Trans. Err. ↓
Cam Rot. Err. (°) ↓
MapAnything
13.53
6.84
GTL
10.15
5.07
Appendix
Table 5. Ablation study on camera extraction pipeline. We compare camera parameters directly estimated by MapAnything with those produced by our pipeline under the same training setting. The results show that our camera extraction pipeline provides more reliable supervision, leading to improved camera translation and rotation accuracy.
Current controllable video generation systems often rely on 2D motion trajectories or sparse drag signals for object motion. These controls are ambiguous because the same 2D trajectory can correspond to different 3D motions, especially when the camera and objects move simultaneously. We present Generative Cinematographer (GenCine), a system that lifts a single image into an editable 3D scene scaffold where artists jointly author camera and foreground motion. Artists specify a camera path and move selected foreground regions using local 3D motion handles. Several handles can move different parts of a subject independently, providing a piecewise-rigid approximation to non-rigid motion without a physics simulator or category-specific prior. To communicate these controls to a pretrained video model, we project them into guidance maps. These maps record where the controlled regions appear in each frame, assign each handle a fixed color across frames and encode the current 3D positions of its controlled points in the same world coordinate system as the background. This lets us describe object motion relative to the scene even as the camera moves. For training, we recover controls from the motion observed in real videos and use ground-truth geometry and trajectories from synthetic videos. We train a lightweight guidance branch and LoRA adapters on a pretrained Wan model to follow these controls. Our experiments show consistent camera-relative motion, improved geometric consistency under viewpoint changes, and strong controllability across diverse real-world scenes.
Camera control has been extensively studied in conditioned video generation; however, performing precisely altering the camera trajectories while faithfully preserving the video content remains a challenging task. The mainstream approach to achieving precise camera control is warping a 3D representation according to the target trajectory. However, such methods fail to fully leverage the 3D priors of video diffusion models (VDMs) and often fall into the Inpainting Trap, resulting in subject inconsistency and degraded generation quality. To address this problem, we propose DepthDirector, a video re-rendering framework with precise camera controllability. By leveraging the depth video from explicit 3D representation as camera-control guidance, our method can faithfully reproduce the dynamic scene of an input video under novel camera trajectories. Specifically, we design a View-Content Dual-Stream Condition mechanism that injects both the source video and the warped depth sequence rendered under the target viewpoint into the pretrained video generation model. This geometric guidance signal enables VDMs to comprehend camera movements and leverage their 3D understanding capabilities, thereby facilitating precise camera control and consistent content generation. Next, we introduce a lightweight LoRA-based video diffusion adapter to train our framework, fully preserving the knowledge priors of VDMs. Additionally, we construct a large-scale multi-camera synchronized dataset named MultiCam-WarpData using Unreal Engine 5, containing 8K videos across 1K dynamic scenes. Extensive experiments show that DepthDirector outperforms existing methods in both camera controllability and visual quality. Our code and dataset will be publicly available.
Dong-Yu Chen, Yixin Guo, Shuojin Yang +2
BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China
Camera-controllable video generation aims to synthesize videos with flexible and physically plausible camera movements. However, existing methods either provide imprecise camera control from text prompts or rely on labor-intensive manual camera trajectory parameters, limiting their use in automated scenarios. To address these issues, we propose a novel Vision-Language-Camera model, termed CT-1 (Camera Transformer 1), a specialized model designed to transfer spatial reasoning knowledge to video generation by accurately estimating camera trajectories. Built upon vision-language modules and a Diffusion Transformer model, CT-1 employs a Wavelet-based Regularization Loss in the frequency domain to effectively learn complex camera trajectory distributions. These trajectories are integrated into a video diffusion model to enable spatially aware camera control that aligns with user intentions. To facilitate the training of CT-1, we design a dedicated data curation pipeline and construct CT-200K, a large-scale dataset containing over 47M frames. Experimental results demonstrate that our framework successfully bridges the gap between spatial reasoning and video synthesis, yielding faithful and high-quality camera-controllable videos and improving camera control accuracy by 25.7% over prior methods.