Hierarchical Diffusion for Camera-controlled Autoregressive Video Generation

Kaihua Chen; Yiming Gong; Deva Ramanan; Tarasha Khurana

cs.CVOct 27, 2026

Hierarchical Diffusion for Camera-controlled Autoregressive Video Generation

Authors: Kaihua Chen, Yiming Gong, Deva Ramanan, Tarasha Khurana

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA · University of Oxford, Oxford, UK

Abstract

We address the task of long-video, camera-controlled generation from a single image. The dominant autoregressive paradigm suffers from two key limitations: (1) error accumulation over time, as distant frames are conditioned on previously generated outputs rather than ground truth, leading to drift; and (2) ambiguity in how to incorporate camera control. Standard solutions to both--such as self-forcing and conditioning on explicit 6-DOF poses--either incur significant computational overhead or produce drifting scene geometries over time. We propose a simple and lightweight framework that combines autoregressive generation (WanAR) with hierarchical long-context denoising (FixAR). To enforce scene geometry remains consistent with input cameras, we draw from feed-forward novel-view synthesis and reformulate video generation as a warping-and-inpainting problem. To mitigate drift, we introduce a hierarchical denoising strategy in which different stages condition on different temporal histories. Specifically, we first generate frames conditioned on the immediate past, and then refine them using a second denoiser conditioned on a temporally subsampled long-range history to enforce global consistency. This fine-to-coarse scheme enables each generated frame to effectively condition on the full temporal context without incurring a prohibitive cost. Experiments on RealEstate10k and DL3DV demonstrate that WanAR+FixAR outperforms prior methods for camera-controlled long-video generation, producing results that are both temporally stable and consistent with input camera trajectories.

Figures & tables

Figure 1: In this work, we consider the problem of camera-controlled video generation. Existing approaches condition on cameras represented as ray encodings or Plucker coordinates, but these learn poses up to a global scale factor. We draw from literature in feed-forward novel-view synthesis that warp given views into future poses and reformulate video generation as video inpainting ( left ). While this allows for precise camera control, minute-long autoregressive video generation in this setup still suffers from drift and error accumulation over time. We propose a hierarchical denoising scheme ( right ) that looks at multiple rollouts from the inpainting stage and corrects for long horizon artifacts.

Figure 2: Overview of WanAR, FixAR, and InterpolatAR. We build upon WAN Team et al. (2025) , a video diffusion model, and adapt it for camera-controlled autoregressive video generation in static 3D scenes. All three stages share the same Wan2.1 backbone weights, with a different stage-specific LoRA checkpoint loaded for each stage. At each rollout, we estimate depth using MoGe-2 Wang et al. (2025c) on the input frame (or the last generated frame from the previous rollout), and warp it into future camera poses to provide geometry-aware conditioning for WanAR ( top ). While WanAR can generate plausible long videos autoregressively, visual drift and artifacts gradually accumulate over multiple rollouts. To mitigate this issue, we introduce a hierarchical denoising framework with a second-stage denoiser, FixAR ( middle ), which refines temporally subsampled keyframes generated across different rollouts using a broader temporal context. Finally, since FixAR only operates on sparse keyframes, we apply InterpolatAR to synthesize intermediate frames between adjacent refined keyframes ( bottom ). The entire three-stage pipeline can itself be rolled out autoregressively to further extend the video. This hierarchical fine-to-coarse formulation enables both accurate camera control and temporally stable long-horizon video generation.

Figure 3: Qualitative comparison on RealEstate10k and DL3DV. We show qualitative results on two test scene from RealEstate10k (top two examples) and DL3DV (bottom two examples) across a range of video generation, and novel-view synthesis baselines. We see that proposed WanAR + FixAR combination of models “renders” the scene using scale-consistent camera poses (as compared to ray encoding in DFoT or Plucker conditioning in LingBot), and show that hierarchical denoising mitigates drift over time which some autoregressive baselines like VMem struggle with. We also note that TrajectoryCrafter, a method designed for novel-view synthesis (in short sequences) is also surprisingly strong at the task of camera-controlled minute-long video generation.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	FVD $\downarrow$	mPSNR $\uparrow$
ViewCrafter Yu et al. (2025c)	12.45	0.361	0.499	67.36	0.0267	595.25	13.44
TrajCrafter Yu et al. (2025b)	14.35	0.428	0.410	30.54	0.0042	347.75	15.54
CogNVS Chen et al. (2025b)	13.22	0.379	0.474	48.58	0.0116	457.35	14.39
VACE Jiang et al. (2025)	15.23	0.487	0.426	49.62	0.0119	393.30	16.89
Lingbot-World Team et al. (2026b)	12.90	0.362	0.513	25.46	0.0012	398.31	13.36
VMem Li et al. (2025b)	12.13	0.336	0.583	49.12	0.0099	465.13	12.46

Table 1: Comparison to state-of-the-art on RealEstate10K Zhou et al. (2018) . We compare our hierarchical diffusion strategy to state-of-the-art approaches for long video generation and novel-view synthesis of static scenes. Our first finding is that a combination of WanAR + FixAR beats all prior art on RealEstate10K, which we attribute to better camera control in WanAR, and better drift mitigation from FixAR. One surprising finding is that novel-view synthesis approaches like TrajectoryCrafter in fact perform better at this task than many long video generation works. With mPSNR, we show better retention of visible regions particularly driven by precise camera control.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	FVD $\downarrow$	mPSNR $\uparrow$
ViewCrafter Yu et al. (2025c)	13.55	0.307	0.459	74.76	0.0114	847.25	13.19
TrajCrafter Yu et al. (2025b)	13.49	0.306	0.428	53.45	0.0042	703.82	13.93
CogNVS Chen et al. (2025b)	12.70	0.268	0.467	81.09	0.0166	882.71	13.04
VACE Jiang et al. (2025)	13.26	0.302	0.476	92.48	0.0209	781.41	13.77
Lingbot-World Team et al. (2026b)	12.87	0.246	0.457	40.26	0.0013	742.09	13.07
VMem Li et al. (2025b)	12.02	0.232	0.563	80.76	0.0138	867.67	12.13

Table 2: Comparison to state-of-the-art on DL3DV Ling et al. (2024) . We compare our method with prior art on the DL3DV dataset which has more challenging camera motion, like sudden translation changes, and 360 ∘ camera pan. Despite this, our method outperforms baselines by a considerable margin. Once again, mPSNR demonstrates better camera control with our warp-and-inpaint strategy for long video generation.

Figure 4: Qualitative analysis of hierarchical denoising. WanAR exhibits drift over long rollouts, while FixAR mitigates the drift and cleans up accumulated artifacts.

Table 7

Figure 5: Top-k evaluation. We note that generative methods for camera-controlled video generation can produce multi-modal outputs for unseen regions of the scene as camera pans in new direction. We conduct such an evaluation for our method. Our finding is that the performance of our final predictions improves with respect to the ground-truth as more samples are drawn from WanAR (thereby covering the generation of multiple plausible 3D scenes in the future).

Method	Backbone	Resolution	GPU Memory	Run Time	# Diffusion Calls
ViewCrafter	StableVideoDiffusion-2B	$576\times 1024\times 25$	22GB	15min	12
TrajCrafter	CogVideoX-5B	$480\times 720\times 49$	48GB	18min	6
CogNVS	CogVideoX-5B	$480\times 720\times 49$	23GB	16min	6
VACE	Wan2.1-14B	$480\times 832\times 49$	51GB	19min	6
VMem	SEVA-1.3B	$576\times 576\times 8$	6GB	4min	6
DFoT	DiT-XL/2-0.68B	$256\times 256\times 8$	7GB	5min	44

Table 5: Computational cost comparison with baselines. Runtime and peak GPU memory are measured on 280-frame RealEstate10K videos using a single 80GB H100 GPU. We also report our method at

480\times 832\times 49

for a resolution-matched comparison.

Setting	Keyframe Interval	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	FVD $\downarrow$
49-length FixAR	2	16.38	0.508	0.336	34.33	0.0021	424.94
	4	16.78	0.522	0.333	34.06	0.0017	428.02
	6	16.71	0.521	0.338	34.55	0.0018	429.14
25-length FixAR	4	16.11	0.490	0.340	38.02	0.0021	461.49
	8	15.91	0.487	0.351	38.72	0.0022	488.46
	12	15.46	0.471	0.384	40.07	0.0030	561.53

Table 6: Ablation on FixAR keyframe interval and sequence length. We compare 49-frame and 25-frame FixAR variants under different keyframe intervals. The 49-frame model remains stable across its training range, while larger intervals degrade the performance of the 25-frame model.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	FVD $\downarrow$
MoGe2	20.22	0.654	0.177	21.90	0.0013	130.58
Depth Anything v3	20.46	0.666	0.172	21.77	0.0014	129.02
MegaSAM	20.74	0.679	0.170	21.92	0.0015	125.77

Table 7: Comparison of different geometry estimation methods. We initialize WanAR using the first-frame depth estimated by each method and report performance after one rollout. Results are similar across estimators, indicating limited sensitivity to the choice of depth estimator.

Figure 6: We identify a key limitation of existing camera-conditioned methods. When explicit 6-DOF poses are used as conditioning—e.g., via Plücker coordinates or ray maps Song et al. (2025) ; Team et al. (2026b) —it is difficult to enforce a consistent scale between the scene and the camera trajectory. As a result, generated videos may exhibit visual inconsistencies ( top ), reflecting scenes rendered under an arbitrary pose scale. Importantly, standard camera-control metrics Xiao et al. (2025) perform a scale alignment with ground-truth camera poses prior to evaluation. Consequently, even when predicted camera trajectories are incorrect up to a global scale factor, they can still achieve low error ( bottom ), masking this failure mode. Current methods evaluating performance with the PSNR metric circumvent this issue, as they overfit to the pose distribution in a particular datasets, which indicates their limited generalization to other scene or camera trajectories. In our approach, we mitigate this issue by aligning MoGE depth predictions to ground-truth depth for each rollout, and by adopting a warping-and-inpainting formulation for long-video generation. We also adopt novel-view synthesis protocols for evaluation, and instead benchmark methods for camera-control with the masked PSNR (mPSNR) metric.

Camera-Controlled Video Generation

Camera-controlled video generation creates videos that follow specified viewpoints and camera paths while preserving coherent scenes and motion.

See all 143 papers

Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control
Jun 26, 2026Haoyuan Wang, Yabo Chen, Haibin Huang +2Camera-Controlled Video Generation Long-Horizon Video Generation
Institute of Artificial Intelligence, China Telecom (TeleAI)
DCARL: A Divide-and-Conquer Framework for Autoregressive Long-Trajectory Video Generation
Mar 25, 2026Junyi Ouyang, Wenbin Teng, Gonglin Chen +2Long-Horizon Video Generation Autoregressive Video Generation
Institute for Creative Technologies · University of Southern California
HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising
Mar 9, 2026Kai Zou, Dian Zheng, Hongbo Liu +3Autoregressive Video Generation Long-Horizon Video Generation
University of Science and Technology of China · Anhui Province Key Laboratory of Digital Security, USTC · The Chinese University of Hong Kong +2

Jun 26, 2026J/K move · Enter open · S save

Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control

Haoyuan Wang, Yabo Chen, Haibin Huang, Chi Zhang+1

Institute of Artificial Intelligence, China Telecom (TeleAI)

Camera-Controlled Video Generation Long-Horizon Video Generation Memory-Augmented Video Generation Autoregressive Video Generation Pose-Conditioned Video Generation Controllable Video Generation

Open paper AskPDF arXiv

Building interactive world models requires generating realistic videos while maintaining controllable dynamics over long horizons. Autoregressive video generation offers a scalable foundation, but suffers from error accumulation and temporal degradation during extended rollouts. This issue is further amplified under heterogeneous controls such as human motion and camera trajectories, which may interfere and destabilize a pretrained video prior, while existing methods often trade off controllability and visual quality. We propose "Directing the World", a fast autoregressive framework for controllable world-model video generation with compositional human-motion and camera-trajectory control. Our key idea is to decouple control learning while preserving a unified autoregressive video prior. We introduce a Fast-Slow Memory training strategy to stabilize long-horizon rollout learning and improve convergence. For human motion control, we design a t-guided Dynamic Projection mechanism and a refined Motion-CFG strategy, enabling temporally smooth and accurate motion alignment without degrading visual fidelity, and supporting multi-person control.After learning a robust motion prior, we introduce a second-stage camera-trajectory control module to compose human dynamics with viewpoint changes for coherent world exploration. We further construct a large-scale dataset with synchronized video, text, human-motion, and camera-trajectory annotations, organized into motion-centric and camera-centric subsets for decoupled training. Extensive experiments show stable long-horizon generation with precise controllability and high visual quality. See more at https://whydahuzi.github.io/Directing-the-World.github.io/.

Abstract

Camera-Controlled Video Generation

Explore similar work

Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control

DCARL: A Divide-and-Conquer Framework for Autoregressive Long-Trajectory Video Generation

HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising