CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation

Haoyu Zhao; Zihao Zhang; Jiaxi Gu; Haoran Chen; Qingping Zheng; Pin Tang; Yeyin Jin; Yuang Zhang

cs.CVApr 10, 2026

CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation

Authors: Haoyu Zhao, Zihao Zhang, Jiaxi Gu, Haoran Chen, Qingping Zheng, Pin Tang, Yeyin Jin, Yuang Zhang,

Organizations: Fudan University · Tencent · Xiamen University · Shanghai Jiao Tong University

Abstract

Camera-controllable video generation aims to synthesize videos with flexible and physically plausible camera movements. However, existing methods either provide imprecise camera control from text prompts or rely on labor-intensive manual camera trajectory parameters, limiting their use in automated scenarios. To address these issues, we propose a novel Vision-Language-Camera model, termed CT-1 (Camera Transformer 1), a specialized model designed to transfer spatial reasoning knowledge to video generation by accurately estimating camera trajectories. Built upon vision-language modules and a Diffusion Transformer model, CT-1 employs a Wavelet-based Regularization Loss in the frequency domain to effectively learn complex camera trajectory distributions. These trajectories are integrated into a video diffusion model to enable spatially aware camera control that aligns with user intentions. To facilitate the training of CT-1, we design a dedicated data curation pipeline and construct CT-200K, a large-scale dataset containing over 47M frames. Experimental results demonstrate that our framework successfully bridges the gap between spatial reasoning and video synthesis, yielding faithful and high-quality camera-controllable videos and improving camera control accuracy by 25.7% over prior methods.

Figures & tables

Panel of Figure 1 — Figure 1: A high-level overview of CT-1’s architecture, its integration with the video generation model, and comparisons.

Figure 2: Overview of the proposed camera-controllable video generation framework based on the CT-1 model, which includes (a) a vision–language module for semantic embedding, (b) a Diffusion Transformer module for modeling camera trajectory distributions, and (c) controllable generation models that synthesize videos conditioned on the trajectories.

Figure 3: Wavelet analysis of camera trajectories over 1,000 examples, including energy dominance, smoothness, geometric retention, and the trade-off between jitter and trajectory deviation across low- and high-frequency components.

Figure 4: Overview of the CT-200K data curation pipeline, designed to address missing or ambiguous camera-motion annotations in existing datasets and to support annotation for reasoning scenarios.

Inference Type	Method	Dolly-in	Pan Left	Truck Left	Tilt Down	Regular	Complex	Average $\>\uparrow$
Inference Type	Method	Dolly-in	or Right	or Right	or Up	Speed	Motion	Average $\>\uparrow$
Trajectory-input-based using VLM	MotionCtrl [ 31 ]	24.0	31.1	18.5	27.5	30.2	21.8	25.5
Trajectory-input-based using VLM	CameraCtrl [ 9 ]	29.0	36.1	22.6	31.3	34.7	27.0	30.1
Trajectory-input-based using AR	MotionCtrl [ 31 ]	18.0	24.4	14.3	22.5	24.9	17.1	20.2
Trajectory-input-based using AR	CameraCtrl [ 9 ]	22.0	28.9	16.7	25.0	28.1	20.6	23.6
Prompt-input-based w/ PE	LTX-Video [ 8 ]	34.0	58.3	28.6	50.0	56.2	45.7	45.5

Table 1: Zero-shot quantitative results on CameraBench100 for camera trajectory estimation across six typical tasks using the Success Rate metric. PE denotes Prompt Extension for fair comparison, and AR denotes autoregressive trajectory generation.

Method	Aesthetic	Imaging	Motion	Dynamic
Method	Quality $\>\uparrow$	Quality $\>\uparrow$	Smoothness	Degree
LTX-Video [ 8 ]	0.535	0.628	0.972	0.981
CogVideoX [ 35 ]	0.540	0.681	0.985	0.445
Wan2.1 [ 26 ]	0.565	0.685	0.977	0.505
Wan2.2 [ 26 ]	0.548	0.665	0.970	0.810
LTX-Video [ 8 ]	0.542	0.634	0.973	0.980

Table 2: Comparison of generated video quality with baseline methods using VBench. Results w/o PE are in gray.

Figure 5: Qualitative comparison in out-of-distribution (OOD) scenarios with CogVideoX [ 35 ] , LTX-Video [ 8 ] , and Wan2.2 [ 26 ] .

Method	Param.	Imaging	Dynamic	Regular	Complex
Method	Param.	Quality	Degree	Speed	Motion
CT-1 (Base)	33M	0.623	0.674	72.3	64.8
CT-1 (Large)	130M	0.685	0.769	82.7	76.4
CT-1 (Huge)	458M	0.708	0.830	87.6	81.5

Table 3: Ablation of camera control and video generation performance across different CT-1 scales.

Method	Imaging	Dynamic	Regular	Complex
Method	Quality	Degree	Speed	Motion
WavReg / $\beta=0$	0.681	0.802	82.0	76.1
WavReg / $\beta=1.0$	0.678	0.799	80.9	77.2
WavReg / $\beta=0.5$	0.682	0.808	82.0	79.3
WavReg / $\beta=0.3$	0.702	0.824	83.1	80.4
WavReg / $\beta=0.05$	0.697	0.816	84.3	79.3

Table 4: Impact of the WavReg loss and the parameter

\beta

on the performance of the CT-1 model.

Method	Imaging	Dynamic	Regular	Complex
Method	Quality	Degree	Speed	Motion
Prompt $\rightarrow$ Images	0.663	0.780	82.0	75.0
Image $\rightarrow$ Prompts	0.725	0.810	88.8	85.9

Table 5: Ablation analysis under the settings of the same image with different prompts (Image

\rightarrow

Prompts) and the same prompt with different images (Prompt

\rightarrow

Images).

Data	Scale (K)	Imaging	Dynamic	Regular	Complex
Data	Scale (K)	Quality	Degree	Speed	Motion
w/ GS	80 / 120	0.674	0.782	80.9	70.7
w/ RS	80 / 80	0.631	0.735	77.5	73.9
CT-200K	80 / 200	0.691	0.809	84.2	77.8

Table 6: Comparison of CT-1 training performance across different dataset settings (GS: General Scenarios; RS: Reasoning Scenarios).

a/b

denotes an

a

subset from the

b

dataset.

Model	Length	Time cost (seconds)	Memory (GB)
CT-1 model	13-step	22.81	28.91
Video generation	49-frame	429.02	39.48

Table 7: Comparison of the time and memory costs of the CT-1 estimation stage and the video generation stage.

Figure 6: Visualization of camera trajectories estimated by CT-1. The red (X), green (Y), and blue (Z) axes indicate the camera’s local coordinate system [ 24 ] : the red axis denotes the right–left direction, the green axis represents the vertical direction (down or up), and the blue axis corresponds to the viewing direction (forward or backward, i.e., zoom-in or zoom-out).

Figure 7: Visualization of camera trajectories estimated by CT-1 under longer textual descriptions.

Figure 8: Visualization of a single image with multiple camera descriptions. Given a fixed reference image, we apply three different camera motion descriptions to the same scene and visualize the camera trajectories predicted by CT-1.

Figure 9: Visualization of a single camera description with multiple reference images. Given a fixed camera motion description, we pair it with three different reference images and infer the corresponding camera trajectories using CT-1.

Figure 10: Qualitative results of generated videos. Given an input reference image and a textual prompt, CT-1 infers semantically consistent camera motion trajectories, which are further used to drive a camera-controllable video generation model [ 39 ] . In each case, scene-related elements are highlighted with a light blue background to indicate the visual content explicitly referenced by the input camera description within the user intent.

Figure 11: Qualitative results of applying the trajectories results of the CT-1 on CameraCtrl [ 9 ] model, which is a camera-controllable video generation model. The tested scenes are from the RealEstate10K dataset [ 43 ] .

Figure 12: Qualitative results of applying the trajectories results of the CT-1 on MotionCtrl [ 31 ] model, which is a camera-controllable video generation model. The tested scenes are from the MultiCamVideo dataset [ 1 ] .

Figure 13: Qualitative results of deep reasoning-based setting, showcasing the CT-1 model’s ability to infer camera motion from semantic and visual clues in complex scenes. The red boxes highlight implicit cues in the text that require visual context for interpretation.

Figure 14: Qualitative results of camera-controllable video generation in driving scenarios on DrivingDoJo dataset [ 30 ] . For each example, we provide a reference image and a forward-driving textual prompt as input to CT-1, which predicts the corresponding camera trajectory and drives the subsequent video diffusion model for synthesis.

Figure 15: User preference on 30 selected videos. Longer pink bars indicate greater satisfaction with our method.

Method	Aesthetic	Imaging	Regular	Complex
Method	Quality	Quality	Speed	Motion
$\textrm{CT-1}^{*}$ -GR	0.557	0.681	83.9	76.8
$\textrm{CT-1}^{*}$ -AR	0.573	0.688	85.1	79.4
CT-1	0.585	0.709	87.6	81.5

Table 8: Comparison of camera trajectory quality with different CT-1 variants. The second-best results are underlined.

Evaluation Setting	Size	Protocol	CT-1	Wan2.2 [ 26 ] w/ PE	Gap	Stability
Fixed CameraBench100	100	Fixed balanced subset	81.6	64.9	+16.7	–
Bootstrap CI	100	1000 bootstrap resamples	81.6 [74.8, 87.9]	64.9 [56.8, 72.7]	+16.7	CI reported
Random balanced subsets	100	Mean over 20 subsets	80.9 $\pm$ 1.7	63.8 $\pm$ 2.2	+17.1 $\pm$ 1.1	CT-1 wins 20/20
Larger balanced subsets	200	Mean over 10 subsets	81.2 $\pm$ 1.1	64.3 $\pm$ 1.5	+16.9 $\pm$ 0.8	CT-1 wins 10/10
Larger balanced subsets	300	Mean over 10 subsets	81.0 $\pm$ 0.9	64.5 $\pm$ 1.2	+16.5 $\pm$ 0.6	CT-1 wins 10/10

Table 9: Statistical stability analysis of CameraBench100. We compare CT-1 and Wan2.2 across different test sets.

Method	Aesthetic	Imaging	Regular	Complex
Method	Quality	Quality	Speed	Motion
VelReg	0.559	0.683	84.1	76.9
AccReg	0.562	0.705	85.7	79.1
Jerk	0.567	0.702	86.5	80.0
LowPass	0.570	0.696	87.0	80.7
WavReg	0.585	0.709	87.6	81.5

Table 10: Comparison of different trajectory regularization strategies. The second-best results are underlined.

Method	Aesthetic	Imaging	Regular	Complex
Method	Quality	Quality	Speed	Motion
Text-only Modeling	0.453	0.578	41.7	33.9
Vision-only Modeling	0.546	0.671	40.4	32.1
Pooled Modeling	0.579	0.683	82.8	76.2
<CAM> Modeling	0.585	0.709	87.6	81.5

Table 11: Comparison of different camera-context token strategies. The second-best results are underlined.

Figure 16: High- and low-frequency analysis of camera trajectories within a single scene, including angular velocity magnitude, linear velocity magnitude, acceleration energy, and high-frequency D1 energy.

Figure 17: Example instances from the CT-200 dataset. For the raw data, we apply the data curation pipeline proposed in this work to produce fine-grained annotations of camera motion descriptions. The figure shows representative video frames along with their corresponding camera motion prompts.

Camera-Controlled Video Generation

Camera-controlled video generation creates videos that follow specified viewpoints and camera paths while preserving coherent scenes and motion.

See all 142 papers

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation
Jan 15, 2026Dong-Yu Chen, Yixin Guo, Shuojin Yang +2Camera-Controlled Video Generation Video Diffusion Models
BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China
Gen Through the Lens: Camera-Controllable Video Generation via Cross-Coordinate Conditioning Fusion
Oct 11, 2026Shuai Li, Jinxu Xiang, Xinyuan Li +5Camera-Controlled Video Generation Image-to-Video Generation
Tencent Video Solar System
TriMotion: Modality-Agnostic Camera Control for Video Generation
Jun 18, 2026Seunghyun Shin, Jifei Song, Wooseok Jeon +2Camera-Controlled Video Generation Multimodal Learning
GIST · Huawei Noah’s Ark Lab · Yonsei University +1

Jan 15, 2026J/K move · Enter open · S save

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation

Dong-Yu Chen, Yixin Guo, Shuojin Yang, Tai-Jiang Mu+1

BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China

Camera-Controlled Video Generation Video Diffusion Models Controllable Video Generation

Open paper AskPDF arXiv

Camera control has been extensively studied in conditioned video generation; however, performing precisely altering the camera trajectories while faithfully preserving the video content remains a challenging task. The mainstream approach to achieving precise camera control is warping a 3D representation according to the target trajectory. However, such methods fail to fully leverage the 3D priors of video diffusion models (VDMs) and often fall into the Inpainting Trap, resulting in subject inconsistency and degraded generation quality. To address this problem, we propose DepthDirector, a video re-rendering framework with precise camera controllability. By leveraging the depth video from explicit 3D representation as camera-control guidance, our method can faithfully reproduce the dynamic scene of an input video under novel camera trajectories. Specifically, we design a View-Content Dual-Stream Condition mechanism that injects both the source video and the warped depth sequence rendered under the target viewpoint into the pretrained video generation model. This geometric guidance signal enables VDMs to comprehend camera movements and leverage their 3D understanding capabilities, thereby facilitating precise camera control and consistent content generation. Next, we introduce a lightweight LoRA-based video diffusion adapter to train our framework, fully preserving the knowledge priors of VDMs. Additionally, we construct a large-scale multi-camera synchronized dataset named MultiCam-WarpData using Unreal Engine 5, containing 8K videos across 1K dynamic scenes. Extensive experiments show that DepthDirector outperforms existing methods in both camera controllability and visual quality. Our code and dataset will be publicly available.

Abstract

Camera-Controlled Video Generation

Explore similar work

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation

Gen Through the Lens: Camera-Controllable Video Generation via Cross-Coordinate Conditioning Fusion

TriMotion: Modality-Agnostic Camera Control for Video Generation