Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation

Chenjian Gao; Zhihao Hu; Jianqi Ma; Jun Zhang; Weidong Zhang; Tianfan Xue

cs.CVSep 29, 2026

Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation

Authors: Chenjian Gao, Zhihao Hu, Jianqi Ma, Jun Zhang, Weidong Zhang, Tianfan Xue

Organizations: MMLab, The Chinese University of Hong Kong · Tencent AIPD

Abstract

Autoregressive (AR) video diffusion enables low-latency, streamable video generation, but prediction errors often accumulate over long rollouts. Training the generator on its own rollouts exposes it to these imperfect histories. However, existing video-level distribution matching distillation (DMD) scores the whole rollout jointly. Because a chunk is evaluated together with its past and future, its correction can favor matching artifacts in the surrounding context merely to preserve temporal consistency. To provide a clearer visual-quality signal, we introduce Rollout-Marginal Distillation (RMD). RMD retains the generated history for AR prediction but scores each chunk independently against a chunk teacher, ensuring its quality correction is not compromised by an imperfect temporal context. To compensate for the lack of temporal context in independent chunk scoring, RMD subsequently applies video-level DMD to restore temporal coherence. Extensive experiments demonstrate that RMD maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines. Code and video results are available at https://cjeen.github.io/RMD

Figures & tables

Figure 1: Long-horizon autoregressive video generation. Trained on 5-second rollouts, RMD maintains high visual quality over 500-second rollouts ( $100\times$ the training horizon). Notably, generation relies solely on a fully sliding context window, without fixed anchors, additional explicit memory mechanisms, or modifications to the generator architecture.

Figure 2: Comparison of real-score predictions under video-level DMD and RMD. (a) The autoregressive rollout accumulates visual artifacts. (b) Video-level DMD scores the sequence jointly, causing the real-score prediction to retain these historical artifacts. (c) RMD scores the current chunk independently, providing a clean target regardless of the degraded context.

Figure 3: Overview of Rollout-Marginal Training. The AR generator produces a causal rollout, sampling the initial chunk ( $x_{1}$ ) at a random-step exit and continuation chunks ( $x_{2},\dots,x_{T}$ ) at the last-step exit. Adapted score networks evaluate them independently without temporal context. The resulting gradients ( $g_{1},\dots,g_{T}$ ) are backpropagated via differentiable causal replay.

Figure 4: Qualitative comparison of long-horizon generation. While baseline methods suffer from severe color saturation and structural collapse beyond their training horizon, RMD consistently preserves sharp subject details and stable scene structures.

Chunk size	Method	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$
	Self Forcing	70.94	77.89	43.17
1	Causal Forcing	69.88	78.34	36.03
	RMD (ours)	81.26	84.52	68.24
	Self Forcing	77.44	81.99	59.22
3	Causal Forcing	74.88	80.53	52.27
	RMD (ours)	81.48	85.03	67.24

Table 1: Quantitative comparison on long-horizon generation. Evaluated on VBench-Long, RMD consistently outperforms video-level distillation baselines across all metrics.

Figure 5: Performance over increasing rollout lengths. VBench-Long scores are evaluated at durations ranging from 10 to 60 seconds. As generation extends, RMD maintains stable visual quality and significantly mitigates the performance degradation suffered by baseline methods.

Figure 6: Qualitative ablation results. The spatio-temporal slice is extracted along the red line. Removing the rollout-marginal objective ( video DMD only ) causes severe color artifacts. Removing video-level refinement ( marginal only ) introduces temporal flickering (jagged slice artifacts). Removing asymmetric denoising exits ( random exits ) destroys the temporal prior, resulting in severe temporal jitter. Full RMD achieves both high visual quality and temporal coherence.

Variant	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$
video DMD only	72.44	78.68	47.50
marginal only	80.64	84.90	63.59
random exits	78.29	82.48	61.56
RMD (full)	81.26	84.52	68.24

Table 2: Quantitative ablation results. (a) Removing refinement with video DMD ( marginal only ) or asymmetric exits ( random exits ) degrades overall performance. (b) Temporal metrics confirm that this refinement is essential for temporal coherence.

Explore similar work

Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation
Sep 11, 2026Zixuan Duan, Xunzhi Xiang, Yabo Chen +6Autoregressive Video Diffusion Models Autoregressive Video Generation
Nanjing University, Institute of Artificial Intelligence, China Telecom (TeleAI), China · Institute of Artificial Intelligence, China Telecom (TeleAI), China · Fudan University, Institute of Artificial Intelligence, China Telecom (TeleAI), China
From Scores to Samples: Elastic Forcing for Autoregressive Video Generation
Sep 28, 2026Chi Zhang, Yueyi Liu, Shi Haoyang +5Autoregressive Video Generation Distribution Matching Distillation
College of AI, Tsinghua University · IAIR, Xi’an Jiaotong University · Xianghui Academy, Fudan University +1
Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
Sep 8, 2026Zhuoran Zhao, Shengju Qian, Tongtong Liang +7Autoregressive Video Diffusion Models Video Diffusion Models
1HKUST(GZ) · 2HKUST · 3LIGHTSPEED +3

Sep 11, 2026, cs.CVJ/K move · Enter open · S save

Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation

Zixuan Duan, Xunzhi Xiang, Yabo Chen, Xin Zhang+5

Nanjing University, Institute of Artificial Intelligence, China Telecom (TeleAI), China · Institute of Artificial Intelligence, China Telecom (TeleAI), China · Fudan University, Institute of Artificial Intelligence, China Telecom (TeleAI), China

Autoregressive Video Diffusion Models Autoregressive Video Generation Distribution Matching Distillation Diversity Uncertainty

Open paper AskPDF arXiv

Few-step distillation improves the efficiency of autoregressive (AR) video generation, but often causes diversity collapse: under the same prompt, different noise samples tend to produce highly similar videos with weakened motion dynamics. We analyze this degradation in Distribution Matching Distillation (DMD)-distilled AR video generators and find that, in the autoregressive setting, it takes the form of a structured uncertainty collapse: the mode-seeking bias of DMD maps different noise samples to nearly identical first chunks, and the deterministic AR cache then propagates this collapsed state to all subsequent chunks, turning a local loss of stochasticity at the rollout root into a global suppression of temporal variation. Based on this analysis, we propose Uncertainty DMD, a simple uncertainty-injection framework that restores stochasticity at two key stages of AR generation: a timestep perturbation for the first chunk to increase first-chunk diversity, and a stochastic cache-writing mechanism for later chunks to preserve uncertainty in autoregressive conditioning. The method requires no architectural changes and introduces only lightweight perturbation operations. The same perturbation mechanisms are used during both training and inference. Experiments show that Uncertainty DMD consistently improves diversity and motion dynamics while maintaining comparable per-sample visual quality.

Abstract

Explore similar work

Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation

From Scores to Samples: Elastic Forcing for Autoregressive Video Generation

Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout