Gen Through the Lens: Camera-Controllable Video Generation via Cross-Coordinate Conditioning Fusion

Shuai Li; Jinxu Xiang; Xinyuan Li; Hanlei Guo; Xiyang Tan; Alan Zhao; Xiaotong Zhao; Ran Zhang

cs.CVOct 11, 2026

Gen Through the Lens: Camera-Controllable Video Generation via Cross-Coordinate Conditioning Fusion

Authors: Shuai Li, Jinxu Xiang, Xinyuan Li, Hanlei Guo, Xiyang Tan, Alan Zhao, Xiaotong Zhao, Ran Zhang

Organizations: Tencent Video Solar System

Abstract

We present Gen Through the Lens, a camera-controllable image-to-video generation framework that synthesizes videos from a given reference image while enabling realistic and physically consistent camera operations. Unlike previous models that treat camera motion as abstract conditioning or rely on a single coordinate system, our approach explicitly decomposes and fuses control signals across three coordinate spaces: the camera coordinate (modeled by intrinsic parameters), the world coordinate (represented by spatial geometry hints), and the object coordinate (described by 3D bounding boxes of target subjects). Through cross-coordinate conditioning fusion, these heterogeneous cues are unified to guide the generative process, enabling precise control of camera motion, focus, and framing while preserving temporal and spatial coherence. Extensive experiments demonstrate that our method produces controllable and visually consistent results across diverse camera operations, outperforming existing approaches in both realism and editability.

Figures & tables

Figure 1. Overview of Gen Through the Lens , the first image-to-video generation framework that jointly enables controllable camera motion, object motion, and depth-of-field effects. Starting from a single input image, our model supports cinematic camera trajectories, zoom and focus transitions, and stable object dynamics in a physically consistent manner. The lower left shows a modeling software environment for editing camera trajectories, object motion, and depth-of-field information. The second column shows the three complementary conditioning signals, and the right side shows the generated video results, with the model’s key controllable capabilities summarized below.

Figure 2. Overview of the data processing pipeline. Recovered metric-scale camera parameters and human-centered 3D geometry are converted into geometry-aware conditions for camera motion, object motion, and depth-of-field control.

Figure 3. Overview of the proposed framework. The model jointly conditions video generation on camera motion, object motion, and depth-of-field signals across camera, world, and object coordinate spaces.

Method	Cam Trans. Err. $\downarrow$	Cam Rot. Err. (°) $\downarrow$
VGGT	41.25	20.34
Pi3	40.36	17.21
MapAnything	12.39	17.43
Ours	11.74	17.11

Table 1. Metric-scale camera pose accuracy on the MultiCamVideo Dataset with ground-truth camera annotations. Methods that estimate camera poses only up to scale (VGGT and Pi3) exhibit large errors on sequences with significant camera motion. Our method recovers metrically consistent camera parameters and achieves lower errors, validating the effectiveness of the proposed pipeline.

Method	FID $\downarrow$	FVD $\downarrow$	Qwen VQA Score $\uparrow$	Cam Trans. Err. $\downarrow$	Cam Rot. Err. (°) $\downarrow$	mIoU $\uparrow$	BBox Center Err. (%) $\downarrow$	BBox Rot. Err. (°) $\downarrow$	Traj. Err. (%) $\downarrow$
PAC	70.17	172.51	0.56	18.15	8.77	0.52	6.37	41.83	8.68
RC-I2V	55.12	98.76	0.70	9.86	4.63	$0.45^{\ast}$	$7.42^{\ast}$	$47.34^{\ast}$	7.29
GEN3C	64.24	114.33	0.62	11.38	5.02	$0.47^{\ast}$	$8.24^{\ast}$	$46.22^{\ast}$	8.39
GTL 14B	53.71	86.54	0.81	8.59	4.89	0.61	4.40	21.73	6.37

Table 2. Quantitative comparison with state-of-the-art methods. Lower is better (

\downarrow

) for error-based metrics and latent space distance, while higher is better (

\uparrow

) for overlap and semantic metrics. Screen-space errors are reported as percentages of the image diagonal. Metrics marked with

\ast

should be interpreted with caution, as RealCam-I2V and GEN3C do not support explicit control of object motion or target object positioning, which may bias object-related metrics toward worse values.

Figure 4. Qualitative comparison of camera controllability with state-of-the-art methods. Gen3C and RealCam-I2V use identical camera motion extracted from the original video, with the first frame as input. CamCloneMaster and Seedance 2.0 use the original video as the camera reference. Our Camera condition encodes recovered extrinsics and intrinsics. For clarity, text prompts are omitted in this figure and provided in the Appendix Sec. E .

Figure 5. Qualitative comparison of object motion control. We compare our method with Perception-as-Control under aligned camera conditions and comparable object motion controls, focusing on the controllability and stability of foreground object motion. For clarity, text prompts are omitted in this figure and provided in the Appendix Sec. E .

Figure 6. Disentanglement between camera and object motion. Keeping the object fixed in screen space under camera motion requires the object to move together with the camera in 3D. The prompts used here avoid specifying object motion and are provided in Appendix Sec. E .

Table 9

Figure 7. Visualization of depth-of-field conditions and generated results. Our representation enables controllable bokeh strength, time-varying focus shifts, and stable depth-of-field under camera and object motion. Note that focus cues are encoded through subtle pixel-level variations, which may not be visually salient. The corresponding text prompts are provided in Appendix Sec. E .

Figure 8. Hitchcock zoom effect synthesized by our model. In this example, the camera moves backward by 3 meters while the zoom is adjusted accordingly to approximately preserve the projected size of the subject in the image, as indicated by the object motion condition video. The corresponding text prompts are provided in Appendix Sec. E .

Figure 9. Additional qualitative results corresponding to the experiments. Text prompts are omitted in this figure and provided in the Appendix Sec. E .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Camera Extraction Method	Cam Trans. Err. $\downarrow$	Cam Rot. Err. (°) $\downarrow$
MapAnything	13.53	6.84
GTL	10.15	5.07

Appendix

Table 5. Ablation study on camera extraction pipeline. We compare camera parameters directly estimated by MapAnything with those produced by our pipeline under the same training setting. The results show that our camera extraction pipeline provides more reliable supervision, leading to improved camera translation and rotation accuracy.

Camera-Controlled Video Generation

Camera-controlled video generation creates videos that follow specified viewpoints and camera paths while preserving coherent scenes and motion.

See all 142 papers

Generative Cinematographer: Composing Camera and Object Motion in 3D
Oct 1, 2026Jiahan Zhang, Chaohao Yang, Namitha Guruprasad +4Camera-Conditioned Video Generation Camera-Controlled Video Generation
Johns Hopkins University
Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation
Jan 15, 2026Dong-Yu Chen, Yixin Guo, Shuojin Yang +2Camera-Controlled Video Generation Video Diffusion Models
BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China
CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation
Apr 10, 2026Haoyu Zhao, Zihao Zhang, Jiaxi Gu +10Camera-Controlled Video Generation Video Diffusion Models
Fudan University · Tencent · Xiamen University +1

Oct 1, 2026J/K move · Enter open · S save

Generative Cinematographer: Composing Camera and Object Motion in 3D

Jiahan Zhang, Chaohao Yang, Namitha Guruprasad, Vivekjyoti Banerjee+3

Johns Hopkins University

Camera-Conditioned Video Generation Camera-Controlled Video Generation 3D Scene Generation Controllable Video Generation

Open paper AskPDF arXiv

Current controllable video generation systems often rely on 2D motion trajectories or sparse drag signals for object motion. These controls are ambiguous because the same 2D trajectory can correspond to different 3D motions, especially when the camera and objects move simultaneously. We present Generative Cinematographer (GenCine), a system that lifts a single image into an editable 3D scene scaffold where artists jointly author camera and foreground motion. Artists specify a camera path and move selected foreground regions using local 3D motion handles. Several handles can move different parts of a subject independently, providing a piecewise-rigid approximation to non-rigid motion without a physics simulator or category-specific prior. To communicate these controls to a pretrained video model, we project them into guidance maps. These maps record where the controlled regions appear in each frame, assign each handle a fixed color across frames and encode the current 3D positions of its controlled points in the same world coordinate system as the background. This lets us describe object motion relative to the scene even as the camera moves. For training, we recover controls from the motion observed in real videos and use ground-truth geometry and trajectories from synthetic videos. We train a lightweight guidance branch and LoRA adapters on a pretrained Wan model to follow these controls. Our experiments show consistent camera-relative motion, improved geometric consistency under viewpoint changes, and strong controllability across diverse real-world scenes.

Abstract

Camera-Controlled Video Generation

Explore similar work

Generative Cinematographer: Composing Camera and Object Motion in 3D

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation

CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation