LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation

Shengxiang Ji; Boyang Wang; Haiyang Xu; Bingnan Li; Yucheng Mao; Zeyuan Chen; Xiaojun Shan; Xiang Zhang

cs.CVSep 29, 2026

LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation

Authors: Shengxiang Ji, Boyang Wang, Haiyang Xu, Bingnan Li, Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, +4 more

Organizations: UC San Diego · University of Virginia · Meta · Amazon · Lambda

Abstract

We introduce LIFT, a unified image-to-video generation framework that complements camera control with Layout-In-FuTure control, enabling users to specify what should appear in a future view and where it should appear. This addresses a practical need in controllable video generation: given an initial image, users often care not only about how the camera moves, but also about what the scene should look like at key future moments, especially the final frame. Existing camera controls specify viewpoint trajectories, while text prompts provide only coarse semantic guidance; neither precisely determines the content and spatial layout of future views. This limitation becomes particularly pronounced under large viewpoint changes, where the camera reveals regions that are not visible in the first frame. LIFT therefore uses the last-frame layout as an explicit control signal for the desired future scene. Since learning from such sparse layout guidance is substantially more challenging than conditioning on dense per-frame layouts, we introduce on-policy self-distillation (OPSD) to transfer the control capability of a dense-layout teacher to a last-frame-layout student. We further curate LIFT-Vista, a dataset featuring large viewpoint changes with camera and temporally consistent layout annotations. Experiments show that LIFT improves video quality, future-layout controllability, and camera controllability over other methods.

Figures & tables

Figure 2: Data Curation Pipeline. (a) World-exploration video collection and metadata filtering. (b) Clip selection based on translation distance, FoV expansion, and content change. (c) Annotation of camera trajectories, spatiotemporal layouts, and captions.

Figure 3: Model Architecture. Layout latent is channel-concatenated with the noisy video latent and first-frame latent. Camera tokens are injected into the DiT stream through token-wise addition. Color-referenced layout local prompts are appended to the global caption.

Figure 4: Dual-mode OPSD. The student alternates between last-frame-layout and camera-only modes and distills selected rollout states from a frozen dense-layout teacher with a flow-matching anchor loss. The student and teacher are both initialized from $\theta_{\mathcal{D}}$ .

Figure 5: Qualitative comparison. Bounding boxes indicate the target locations of conditioned objects. MagicMotion lacks explicit camera control and often produces inconsistent scene evolution or incorrect objects, while Uni3C follows the prescribed camera trajectory but leaves newly revealed regions uncontrolled, resulting in unspecified contents, e.g., a pillar in (b) and a car street in (d). In contrast, ours follows the prescribed camera motion and realizes the specified future-view layout.

Method	Control			Video Quality			Camera Error		Semantic Consistency
Method	Camera	Object Motion	Future Layout	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$
Direct-a-Video	✓	✓	✗	539.28	71.56	0.80	28.48	2.28	0.10	0.11	0.14
MagicMotion	✗	✓	✗	277.94	18.32	0.57	15.74	1.59	0.41	0.53	0.21
GEN3C	✓	✗	✗	89.59	15.60	0.51	3.91	2.62	0.18	0.38	0.19
Uni3C	✓	✗	✗	111.60	12.75	0.42	3.23	0.70	0.31	0.47	0.21
LIFT	✓	✓	✓	99.35	12.84	0.42	2.97	0.59	0.51	0.59	0.24

Table 1: Quantitative comparison with state-of-the-art controllable video generation methods. We report video quality, camera trajectory accuracy, and semantic consistency metrics. The best , second-best , and third-best results are highlighted accordingly.

Method	Training Sample (Steps $\times$ BS)	Video Quality			Camera Error		Semantic Consistency
Method	Training Sample (Steps $\times$ BS)	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$
Direct last-frame SFT	$4\mathrm{K}\times 32=128\mathrm{K}$	114.03	12.96	0.43	2.98	0.56	0.44	0.56	0.23
D2S-SFT	$4\mathrm{K}\times 32=128\mathrm{K}$	110.53	12.99	0.42	3.02	0.51	0.47	0.59	0.23
Ours	$500\times 16=8\mathrm{K}$	99.35	12.84	0.42	2.97	0.59	0.51	0.59	0.24
Ours w/o SFT anchor	–	129.14	17.16	0.45	3.36	0.68	0.49	0.57	0.22

Table 2: Ablation study of SFT and OPSD. D2S-SFT denotes dense-to-sparse curriculum SFT. Training cost is measured by the number of training sample updates (steps

\times

global batch size). OPSD achieves the best results with much fewer training samples of the SFT-based alternatives.

OPSD Variant	Lastframe-Layout Mode							Camera-Only Mode
OPSD Variant	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	$\mathrm{mIoU}$ $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$
Teacher (Dense layout)	123.50	13.72	2.92	0.51	0.60	0.60	0.23	147.20	14.25	3.85	0.63
Student @ step0	137.10	14.08	3.58	0.58	0.40	0.56	0.22	147.20	14.25	3.85	0.63
Lastframe-layout single-mode	102.07	13.07	2.88	0.58	0.49	0.59	0.23	121.88	13.42	3.09	0.67
Camera-only single-mode	108.30	14.00	3.04	0.59	0.38	0.57	0.22	105.22	13.77	3.03	0.63
Ours (dual-mode)	99.35	12.84	2.97	0.59	0.51	0.59	0.24	95.66	13.18	3.30	0.63

Table 3: Ablation study of dual-mode OPSD. The dense-layout teacher and the student before OPSD training are included as references.

$\mathbf{p_{\mathrm{layout}}}$	Lastframe-Layout Mode							Camera-Only Mode
$\mathbf{p_{\mathrm{layout}}}$	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$
0.5	102.63	13.40	2.936	0.620	0.4898	0.5862	0.2306	106.01	13.85	3.190	0.703
0.7	99.35	12.84	2.968	0.593	0.5094	0.5861	0.2351	95.66	13.18	3.304	0.632
0.9	93.05	12.96	2.978	0.653	0.5060	0.5832	0.2321	98.62	13.34	3.357	0.757

Table 4: Ablation study of mode sampling probability in dual-mode OPSD.

p_{\mathrm{layout}}

denotes the probability of sampling the lastframe-layout mode during training.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Visualization of data distribution. (a, b) Distributions of the camera-motion metrics over all candidate windows before filtering (blue) and over the final layout training set (red). Filtering removes the mass of near-static windows and shifts both metrics toward larger viewpoint changes. (c) Joint distribution of the two camera-motion metrics in the training set, with 50% and 90% mass contours per source dataset. (d) Joint distribution of the two content change ratios. The two directions are correlated and complementary but not redundant. (e) Number of training clips per dataset.

Setting	Stage 1	Stage 2	Stage 3
Training mode	Camera Control (SFT)	Dense Layout Control (SFT)	Dual-Mode OPSD
Base Model	Wan2.1-Fun-V1.1-1.3B- Control-Camera	Stage 1	Stage 2
Dataset size	120,898	58,272	58,272
Training steps	8,000	4,000	500
Global batch size	32	32	16
Learning rate	$1\times 10^{-5}$	$1\times 10^{-4}$	$5\times 10^{-5}$

Appendix

Table 5: Training configuration of LIFT across the three stages.

Layout given at	Video Quality			Camera Error		Semantic Consistency
Layout given at	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$
Dense	98.84	12.58	0.398	2.544	0.594	0.6217	0.6081	0.2403
8 frames	92.72	12.86	0.406	2.772	0.574	0.5758	0.5907	0.2384
4 frames	95.86	12.79	0.407	2.751	0.559	0.5764	0.5859	0.2379
Last frame only †	99.35	12.84	0.418	2.968	0.593	0.5094	0.5861	0.2351

Appendix

Table 6: Comparisons of different layout sparsity. We compare four layout sparsities without retraining: layou bboxes on all 81 frames, on 8 frames (10, 20, …, 70, 81), on 4 frames (20, 40, 60, 81), and on the last frame only. † marks our target sparsity.

Setting	Direct LF-SFT	D2S-SFT	OPSD (Ours)
Starting checkpoint	Stage 1	Stage 1	Stage 1
Training schedule	8K LF	4K Dense + 2K 8-frame + 2K LF	4K Dense + 500 OPSD
Global batch size	32	32	32 (Dense) / 16 (OPSD)
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$ (Dense) / $5\times 10^{-5}$ (OPSD)
Total training steps	8,000	8,000	4,500
Sample updates (Steps $\times$ BS)	256K	256K	136K

Appendix

Table 7: Implementation details for the SFT and OPSD ablations. Direct LF-SFT denotes direct last-frame SFT. D2S-SFT denotes dense-to-sparse curriculum SFT. For a fair comparison, all three methods are counted from the same Stage 1 camera-control checkpoint, and the dense-layout SFT stage required before OPSD is included in the training cost of our method. Sample updates are computed as training steps

\times

global batch size over the full adaptation process.

Figure 7: SFT and OPSD Comparison. (a) Direct last-frame SFT largely ignores the sparse layout condition, leaving the target sedan absent until the final frame, while dense-to-sparse SFT introduces it too early and with inaccurate spatial alignment. In contrast, OPSD produces a trajectory that better matches the ground-truth evolution. (b) Direct last-frame SFT fails to realize several conditioned objects, whereas dense-to-sparse SFT improves object presence but still exhibits poor temporal alignment before the last frame. OPSD more faithfully follows both the target layout and its temporal evolution.

Figure 8: Dual-mode OPSD Comparison. The last-frame-layout mode uses both the camera trajectory and last-frame layout, whereas the camera-only mode uses only the camera trajectory. Last-frame-layout single-mode OPSD follows the prescribed layout, but yields lower video quality than dual-mode OPSD in camera-only inference, while camera-only single-mode OPSD preserves camera control but often fails to realize the specified future layout. In contrast, dual-mode OPSD maintains strong camera control in both inference settings while faithfully following the future-layout condition when provided.

Figure 9: Student–teacher comparison in dense-to-lastframe layout OPSD. The labels $t=0,2,\ldots,12$ denote denoising-step indices rather than the continuous diffusion time used in the equations. Within the first 10 denoising steps, the dense-layout teacher provides substantial corrections on the student-visited states, producing results that better conform to the specified layout. Afterwards, the teacher correction becomes much weaker, motivating us to concentrate OPSD supervision on the first 10 high-noise states.

Method	Video Quality			Camera Error		Semantic Consistency			Time
Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$	s/step $\downarrow$
All 50 rollout states	127.24	15.02	0.441	3.467	0.624	0.4786	0.5740	0.2234	544.82
Equal 10 states	120.23	15.17	0.439	3.228	0.589	0.4807	0.5777	0.2223	185.28
Late 10 states	197.63	17.82	0.494	6.347	0.906	0.3089	0.5227	0.2024	183.56
Prefix 10 states	127.02	17.77	0.442	2.961	0.544	0.4922	0.5664	0.2219	117.21
Prefix 10 states + SFT anchor	102.07	13.07	0.422	2.879	0.583	0.4934	0.5883	0.2313	125.20

Appendix

Table 8: Ablation of OPSD state selection.

Figure 10: Comparison of different ODE state sampling in OPSD. Distilling only on the late 10 states (the third row) yields the weakest layout following, indicating that global layout structure is mainly determined during the early high-noise stage. Prefix-state distillation provides stronger layout control, while adding the SFT anchor further prevents drifting and visual-quality degradation.

Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$
w/o filter	278.85	32.88	0.52	15.62	4.05
w/ filter	266.58	30.99	0.51	13.14	3.87

Appendix

Table 9: Ablation of the proposed camera-motion filtering strategy. We compare models trained on randomly sampled and filtered data under the same training configuration. The proposed filtering strategy consistently improves both visual quality and camera-control accuracy.

Figure 11: Comparison of the proposed camera-motion filtering strategy. The first row shows results from the model trained on randomly sampled clips, exhibiting degraded visual quality and less stable object appearance. The second row shows results from the model trained on filtered dynamic clips, with better preservation of scene structure and object appearance under camera motion. The third row shows ground-truth frames.

	Ours	Uni3C	MagicMotion
Preference	65.96%	22.34%	11.70%

Appendix

Table 10: User study on preference rates among different camera-controllable video generation methods. Participants select the result that best satisfies interactive editing requirements.

Explore similar work

Dec 9, 2025cs.CV

GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

Recent progress in text-to-video generation has achieved remarkable realism, yet fine-grained control over camera motion and orientation remains elusive, especially with extreme trajectories (e.g., a 180-degree turnaround, or looking directly up or down). Existing approaches typically encode camera trajectories using relative or ambiguous representations, limiting precise geometric control and offering limited support for large rotations. We introduce GimbalDiffusion, a framework that enables camera control grounded in physical-world coordinates, using gravity as a global reference. Instead of describing motion relative to previous frames, our method defines camera trajectories in an absolute coordinate system, allowing accurate, interpretable control over camera parameters. Using panoramic 360-degree videos for training, we cover the full sphere of possible viewpoints, including combinations of extreme pitch and roll that are out-of-distribution of conventional video data. To improve camera control, we introduce null-pitch conditioning, a strategy that prevents the model from overriding camera specifications in the presence of conflicting prompt content (e.g., generating grass while the camera points toward the sky). Finally, we propose new benchmarks to evaluate gravity-aware camera-controlled video generation, assessing models' ability to generate extreme camera angles and quantify their input prompt entanglement.

Frédéric Fortier-Chouinard, Yannick Hold-Geoffroy, Valentin Deschaintre +2

Université Laval, Québec, Canada · Adobe, San Jose, USA

Camera Control Camera Trajectories Video Generation Human Motion Generation Diffusion Dynamics

Ask PDF

Sep 21, 2026cs.CV

CoaG: Cylinders on a Grid for Coarse 3D Layout Control in Video Generation

We ask how little geometry a person has to draw to control both where people stand and where the camera moves in a generated video. Our answer is a ground plane and one cylinder per person. A user draws a grid on the ground, places one cylinder where each person should stand, moves the cylinders and the camera over 81 frames, and the model renders a photoreal video in which the people occupy the cylinders' positions, move as the cylinders move, and are seen from the drawn camera. Appearance comes from a text prompt and a background reference image; layout and motion come from the geometry. Because no dataset pairs such a signal with video, we build the pairs ourselves: an automatic engine writes 2000 captions from a combinatorial seed, generates a clip for each with a text-to-video model, and lifts every clip back to its geometry with person tracking, background inpainting, an agentic ground-mask loop, feed-forward multi-view reconstruction and a plane fit, with no real footage and no manual labels. A LoRA on Wan2.2-Fun-Control trained on 1935 such tuples follows drawn layouts and camera paths on hold-out clips: the generated people match the cylinders' count, order, position and height, the text changes who they are, the reference image changes where they are, and dolly-in, orbit, pan and crane paths are followed, dolly-out only weakly.

Zhangsihao Yang, Mengyi Shan

University of Washington

Video Generation Human Motion Generation 3D Layout Generation Motion Grid Agent Loop

Ask PDF

Jan 15, 2026cs.CV

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation

Camera control has been extensively studied in conditioned video generation; however, performing precisely altering the camera trajectories while faithfully preserving the video content remains a challenging task. The mainstream approach to achieving precise camera control is warping a 3D representation according to the target trajectory. However, such methods fail to fully leverage the 3D priors of video diffusion models (VDMs) and often fall into the Inpainting Trap, resulting in subject inconsistency and degraded generation quality. To address this problem, we propose DepthDirector, a video re-rendering framework with precise camera controllability. By leveraging the depth video from explicit 3D representation as camera-control guidance, our method can faithfully reproduce the dynamic scene of an input video under novel camera trajectories. Specifically, we design a View-Content Dual-Stream Condition mechanism that injects both the source video and the warped depth sequence rendered under the target viewpoint into the pretrained video generation model. This geometric guidance signal enables VDMs to comprehend camera movements and leverage their 3D understanding capabilities, thereby facilitating precise camera control and consistent content generation. Next, we introduce a lightweight LoRA-based video diffusion adapter to train our framework, fully preserving the knowledge priors of VDMs. Additionally, we construct a large-scale multi-camera synchronized dataset named MultiCam-WarpData using Unreal Engine 5, containing 8K videos across 1K dynamic scenes. Extensive experiments show that DepthDirector outperforms existing methods in both camera controllability and visual quality. Our code and dataset will be publicly available.

Dong-Yu Chen, Yixin Guo, Shuojin Yang +2

BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China

Camera Control Video Generation Dramadirector 3D Priors 3D Representation

Ask PDF