LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation

Shengxiang Ji; Boyang Wang; Haiyang Xu; Bingnan Li; Yucheng Mao; Zeyuan Chen; Xiaojun Shan; Xiang Zhang

cs.CVSep 29, 2026

LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation

Authors: Shengxiang Ji, Boyang Wang, Haiyang Xu, Bingnan Li, Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, +4 more

Organizations: UC San Diego · University of Virginia · Meta · Amazon · Lambda

Abstract

We introduce LIFT, a unified image-to-video generation framework that complements camera control with Layout-In-FuTure control, enabling users to specify what should appear in a future view and where it should appear. This addresses a practical need in controllable video generation: given an initial image, users often care not only about how the camera moves, but also about what the scene should look like at key future moments, especially the final frame. Existing camera controls specify viewpoint trajectories, while text prompts provide only coarse semantic guidance; neither precisely determines the content and spatial layout of future views. This limitation becomes particularly pronounced under large viewpoint changes, where the camera reveals regions that are not visible in the first frame. LIFT therefore uses the last-frame layout as an explicit control signal for the desired future scene. Since learning from such sparse layout guidance is substantially more challenging than conditioning on dense per-frame layouts, we introduce on-policy self-distillation (OPSD) to transfer the control capability of a dense-layout teacher to a last-frame-layout student. We further curate LIFT-Vista, a dataset featuring large viewpoint changes with camera and temporally consistent layout annotations. Experiments show that LIFT improves video quality, future-layout controllability, and camera controllability over other methods.

Figures & tables

Figure 2: Data Curation Pipeline. (a) World-exploration video collection and metadata filtering. (b) Clip selection based on translation distance, FoV expansion, and content change. (c) Annotation of camera trajectories, spatiotemporal layouts, and captions.

Figure 3: Model Architecture. Layout latent is channel-concatenated with the noisy video latent and first-frame latent. Camera tokens are injected into the DiT stream through token-wise addition. Color-referenced layout local prompts are appended to the global caption.

Figure 4: Dual-mode OPSD. The student alternates between last-frame-layout and camera-only modes and distills selected rollout states from a frozen dense-layout teacher with a flow-matching anchor loss. The student and teacher are both initialized from $\theta_{\mathcal{D}}$ .

Figure 5: Qualitative comparison. Bounding boxes indicate the target locations of conditioned objects. MagicMotion lacks explicit camera control and often produces inconsistent scene evolution or incorrect objects, while Uni3C follows the prescribed camera trajectory but leaves newly revealed regions uncontrolled, resulting in unspecified contents, e.g., a pillar in (b) and a car street in (d). In contrast, ours follows the prescribed camera motion and realizes the specified future-view layout.

Method	Control			Video Quality			Camera Error		Semantic Consistency
Method	Camera	Object Motion	Future Layout	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$
Direct-a-Video	✓	✓	✗	539.28	71.56	0.80	28.48	2.28	0.10	0.11	0.14
MagicMotion	✗	✓	✗	277.94	18.32	0.57	15.74	1.59	0.41	0.53	0.21
GEN3C	✓	✗	✗	89.59	15.60	0.51	3.91	2.62	0.18	0.38	0.19
Uni3C	✓	✗	✗	111.60	12.75	0.42	3.23	0.70	0.31	0.47	0.21
LIFT	✓	✓	✓	99.35	12.84	0.42	2.97	0.59	0.51	0.59	0.24

Table 1: Quantitative comparison with state-of-the-art controllable video generation methods. We report video quality, camera trajectory accuracy, and semantic consistency metrics. The best , second-best , and third-best results are highlighted accordingly.

Method	Training Sample (Steps $\times$ BS)	Video Quality			Camera Error		Semantic Consistency
Method	Training Sample (Steps $\times$ BS)	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$
Direct last-frame SFT	$4\mathrm{K}\times 32=128\mathrm{K}$	114.03	12.96	0.43	2.98	0.56	0.44	0.56	0.23
D2S-SFT	$4\mathrm{K}\times 32=128\mathrm{K}$	110.53	12.99	0.42	3.02	0.51	0.47	0.59	0.23
Ours	$500\times 16=8\mathrm{K}$	99.35	12.84	0.42	2.97	0.59	0.51	0.59	0.24
Ours w/o SFT anchor	–	129.14	17.16	0.45	3.36	0.68	0.49	0.57	0.22

Table 2: Ablation study of SFT and OPSD. D2S-SFT denotes dense-to-sparse curriculum SFT. Training cost is measured by the number of training sample updates (steps

\times

global batch size). OPSD achieves the best results with much fewer training samples of the SFT-based alternatives.

OPSD Variant	Lastframe-Layout Mode							Camera-Only Mode
OPSD Variant	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	$\mathrm{mIoU}$ $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$
Teacher (Dense layout)	123.50	13.72	2.92	0.51	0.60	0.60	0.23	147.20	14.25	3.85	0.63
Student @ step0	137.10	14.08	3.58	0.58	0.40	0.56	0.22	147.20	14.25	3.85	0.63
Lastframe-layout single-mode	102.07	13.07	2.88	0.58	0.49	0.59	0.23	121.88	13.42	3.09	0.67
Camera-only single-mode	108.30	14.00	3.04	0.59	0.38	0.57	0.22	105.22	13.77	3.03	0.63
Ours (dual-mode)	99.35	12.84	2.97	0.59	0.51	0.59	0.24	95.66	13.18	3.30	0.63

Table 3: Ablation study of dual-mode OPSD. The dense-layout teacher and the student before OPSD training are included as references.

$\mathbf{p_{\mathrm{layout}}}$	Lastframe-Layout Mode							Camera-Only Mode
$\mathbf{p_{\mathrm{layout}}}$	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$	FVD $\downarrow$	FID $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$
0.5	102.63	13.40	2.936	0.620	0.4898	0.5862	0.2306	106.01	13.85	3.190	0.703
0.7	99.35	12.84	2.968	0.593	0.5094	0.5861	0.2351	95.66	13.18	3.304	0.632
0.9	93.05	12.96	2.978	0.653	0.5060	0.5832	0.2321	98.62	13.34	3.357	0.757

Table 4: Ablation study of mode sampling probability in dual-mode OPSD.

p_{\mathrm{layout}}

denotes the probability of sampling the lastframe-layout mode during training.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Visualization of data distribution. (a, b) Distributions of the camera-motion metrics over all candidate windows before filtering (blue) and over the final layout training set (red). Filtering removes the mass of near-static windows and shifts both metrics toward larger viewpoint changes. (c) Joint distribution of the two camera-motion metrics in the training set, with 50% and 90% mass contours per source dataset. (d) Joint distribution of the two content change ratios. The two directions are correlated and complementary but not redundant. (e) Number of training clips per dataset.

Setting	Stage 1	Stage 2	Stage 3
Training mode	Camera Control (SFT)	Dense Layout Control (SFT)	Dual-Mode OPSD
Base Model	Wan2.1-Fun-V1.1-1.3B- Control-Camera	Stage 1	Stage 2
Dataset size	120,898	58,272	58,272
Training steps	8,000	4,000	500
Global batch size	32	32	16
Learning rate	$1\times 10^{-5}$	$1\times 10^{-4}$	$5\times 10^{-5}$

Appendix

Table 5: Training configuration of LIFT across the three stages.

Layout given at	Video Quality			Camera Error		Semantic Consistency
Layout given at	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$
Dense	98.84	12.58	0.398	2.544	0.594	0.6217	0.6081	0.2403
8 frames	92.72	12.86	0.406	2.772	0.574	0.5758	0.5907	0.2384
4 frames	95.86	12.79	0.407	2.751	0.559	0.5764	0.5859	0.2379
Last frame only †	99.35	12.84	0.418	2.968	0.593	0.5094	0.5861	0.2351

Appendix

Table 6: Comparisons of different layout sparsity. We compare four layout sparsities without retraining: layou bboxes on all 81 frames, on 8 frames (10, 20, …, 70, 81), on 4 frames (20, 40, 60, 81), and on the last frame only. † marks our target sparsity.

Setting	Direct LF-SFT	D2S-SFT	OPSD (Ours)
Starting checkpoint	Stage 1	Stage 1	Stage 1
Training schedule	8K LF	4K Dense + 2K 8-frame + 2K LF	4K Dense + 500 OPSD
Global batch size	32	32	32 (Dense) / 16 (OPSD)
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$ (Dense) / $5\times 10^{-5}$ (OPSD)
Total training steps	8,000	8,000	4,500
Sample updates (Steps $\times$ BS)	256K	256K	136K

Appendix

Table 7: Implementation details for the SFT and OPSD ablations. Direct LF-SFT denotes direct last-frame SFT. D2S-SFT denotes dense-to-sparse curriculum SFT. For a fair comparison, all three methods are counted from the same Stage 1 camera-control checkpoint, and the dense-layout SFT stage required before OPSD is included in the training cost of our method. Sample updates are computed as training steps

\times

global batch size over the full adaptation process.

Figure 7: SFT and OPSD Comparison. (a) Direct last-frame SFT largely ignores the sparse layout condition, leaving the target sedan absent until the final frame, while dense-to-sparse SFT introduces it too early and with inaccurate spatial alignment. In contrast, OPSD produces a trajectory that better matches the ground-truth evolution. (b) Direct last-frame SFT fails to realize several conditioned objects, whereas dense-to-sparse SFT improves object presence but still exhibits poor temporal alignment before the last frame. OPSD more faithfully follows both the target layout and its temporal evolution.

Figure 8: Dual-mode OPSD Comparison. The last-frame-layout mode uses both the camera trajectory and last-frame layout, whereas the camera-only mode uses only the camera trajectory. Last-frame-layout single-mode OPSD follows the prescribed layout, but yields lower video quality than dual-mode OPSD in camera-only inference, while camera-only single-mode OPSD preserves camera control but often fails to realize the specified future layout. In contrast, dual-mode OPSD maintains strong camera control in both inference settings while faithfully following the future-layout condition when provided.

Figure 9: Student–teacher comparison in dense-to-lastframe layout OPSD. The labels $t=0,2,\ldots,12$ denote denoising-step indices rather than the continuous diffusion time used in the equations. Within the first 10 denoising steps, the dense-layout teacher provides substantial corrections on the student-visited states, producing results that better conform to the specified layout. Afterwards, the teacher correction becomes much weaker, motivating us to concentrate OPSD supervision on the first 10 high-noise states.

Method	Video Quality			Camera Error		Semantic Consistency			Time
Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$	mIoU $\uparrow$	$SR_{e}$ $\uparrow$	$\mathrm{CLIP}_{\mathrm{local}}$ $\uparrow$	s/step $\downarrow$
All 50 rollout states	127.24	15.02	0.441	3.467	0.624	0.4786	0.5740	0.2234	544.82
Equal 10 states	120.23	15.17	0.439	3.228	0.589	0.4807	0.5777	0.2223	185.28
Late 10 states	197.63	17.82	0.494	6.347	0.906	0.3089	0.5227	0.2024	183.56
Prefix 10 states	127.02	17.77	0.442	2.961	0.544	0.4922	0.5664	0.2219	117.21
Prefix 10 states + SFT anchor	102.07	13.07	0.422	2.879	0.583	0.4934	0.5883	0.2313	125.20

Appendix

Table 8: Ablation of OPSD state selection.

Figure 10: Comparison of different ODE state sampling in OPSD. Distilling only on the late 10 states (the third row) yields the weakest layout following, indicating that global layout structure is mainly determined during the early high-noise stage. Prefix-state distillation provides stronger layout control, while adding the SFT anchor further prevents drifting and visual-quality degradation.

Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	RotErr $\downarrow$	TransErr $\downarrow$
w/o filter	278.85	32.88	0.52	15.62	4.05
w/ filter	266.58	30.99	0.51	13.14	3.87

Appendix

Table 9: Ablation of the proposed camera-motion filtering strategy. We compare models trained on randomly sampled and filtered data under the same training configuration. The proposed filtering strategy consistently improves both visual quality and camera-control accuracy.

Figure 11: Comparison of the proposed camera-motion filtering strategy. The first row shows results from the model trained on randomly sampled clips, exhibiting degraded visual quality and less stable object appearance. The second row shows results from the model trained on filtered dynamic clips, with better preservation of scene structure and object appearance under camera motion. The third row shows ground-truth frames.

	Ours	Uni3C	MagicMotion
Preference	65.96%	22.34%	11.70%

Appendix

Table 10: User study on preference rates among different camera-controllable video generation methods. Participants select the result that best satisfies interactive editing requirements.

Explore similar work

GimbalDiffusion: Gravity-Aware Camera Control for Video Generation
Dec 9, 2025Frédéric Fortier-Chouinard, Yannick Hold-Geoffroy, Valentin Deschaintre +2Camera Control Camera Trajectories
Université Laval, Québec, Canada · Adobe, San Jose, USA
CoaG: Cylinders on a Grid for Coarse 3D Layout Control in Video Generation
Sep 21, 2026Zhangsihao Yang, Mengyi ShanVideo Generation Human Motion Generation
University of Washington
Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation
Jan 15, 2026Dong-Yu Chen, Yixin Guo, Shuojin Yang +2Camera Control Video Generation
BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China

Dec 9, 2025, cs.CVJ/K move · Enter open · S save

GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

Frédéric Fortier-Chouinard, Yannick Hold-Geoffroy, Valentin Deschaintre, Matheus Gadelha+1

Université Laval, Québec, Canada · Adobe, San Jose, USA

Camera Control Camera Trajectories Video Generation Human Motion Generation Diffusion Dynamics

Open paper AskPDF arXiv

Recent progress in text-to-video generation has achieved remarkable realism, yet fine-grained control over camera motion and orientation remains elusive, especially with extreme trajectories (e.g., a 180-degree turnaround, or looking directly up or down). Existing approaches typically encode camera trajectories using relative or ambiguous representations, limiting precise geometric control and offering limited support for large rotations. We introduce GimbalDiffusion, a framework that enables camera control grounded in physical-world coordinates, using gravity as a global reference. Instead of describing motion relative to previous frames, our method defines camera trajectories in an absolute coordinate system, allowing accurate, interpretable control over camera parameters. Using panoramic 360-degree videos for training, we cover the full sphere of possible viewpoints, including combinations of extreme pitch and roll that are out-of-distribution of conventional video data. To improve camera control, we introduce null-pitch conditioning, a strategy that prevents the model from overriding camera specifications in the presence of conflicting prompt content (e.g., generating grass while the camera points toward the sky). Finally, we propose new benchmarks to evaluate gravity-aware camera-controlled video generation, assessing models' ability to generate extreme camera angles and quantify their input prompt entanglement.

Abstract

Explore similar work

GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

CoaG: Cylinders on a Grid for Coarse 3D Layout Control in Video Generation

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation