Self-Aligned Forcing: Streaming Video Diffusion with Differentiable Noisy History

Weiqiang Wang; Zhuokun Chen; Yusheng Dai; Boying Li; Yi Zhang; Hossein Rahmani; Qiuhong Ke; Jianfei Cai

cs.CVSep 29, 2026

Self-Aligned Forcing: Streaming Video Diffusion with Differentiable Noisy History

Authors: Weiqiang Wang, Zhuokun Chen, Yusheng Dai, Boying Li, Yi Zhang, Hossein Rahmani, Qiuhong Ke, Jianfei Cai

Organizations: Monash University · Vivix AI · Lancaster University

Abstract

Autoregressive video diffusion enables interactive streaming generation, but suffers from error accumulation over long rollouts. Self-rollout training reduces exposure bias, yet finite rollouts leave long-range drift unresolved. We observe that the noise level of the history key-value (K/V) representations trades visual quality against motion, and that restoring gradients through the history aligns causal training far more closely with bidirectional training. Motivated by these observations, we introduce Self-Aligned Forcing (SAF), a training scheme that aligns the history of each block with the noise level of the block being denoised. Specifically, the history is the K/V produced by preceding blocks at the same denoising stage, so all blocks at a stage can be denoised in a single forward pass under a causal mask. This keeps the noisy history differentiable, allowing future losses to optimize how it is encoded. SAF therefore avoids a separate no-gradient rollout and per-block timestep-zero recaching, training up to 1.8x faster than prior methods with lower memory. At inference, SAF achieves the highest single-GPU throughput among existing methods and keeps one history bank per stage for a multi-GPU pipeline, reaching 49.1 FPS on 4 GPUs. Experiments show superior long-horizon generation with a better balance between visual quality and motion. Project page: https://anonymous.4open.science/w/self-aligned-forcing/.

Figures & tables

Figure 1: History stage and gradient flow. The model

f_{\theta}

generates blocks from left to right, and the K/V of earlier blocks form the history of the last block, whose loss

\mathcal{L}

is shown. Superscripts give noise levels (

t_{s}

: current block,

t_{c}

: history,

0

: clean). Self-Forcing (a) and HiAR (b) detach the history, so the loss cannot reach it; Self Gradient Forcing (c) and SAF (d) keep this gradient with the history at

t_{c}=0

and

t_{c}=t_{s}

, respectively, but the former needs an extra no-grad rollout.

Panel of Figure 2 — Figure 2: Two observations motivating differentiable noisy history. (a) A fixed LongLive model with varying history noise level $t_{c}$ ; imaging quality and dynamic degree ( Huang et al., 2024 ) are normalized to their values at $t_{c}=0$ and $t_{c}=750$ , respectively. (b) With all settings identical except whether gradients flow through the history K/V, differentiable history raises the gradient cosine similarity to a matched-weight bidirectional reference from $0.643$ to $0.972$ . Error bars show the range over multiple samples.

Figure 3: Overview of SAF. (a) Rollout is parallel across blocks and sequential only across stages (Section 3.3 ). (b) Stage alignment enables joint denoising with differentiable history in one block-causal forward pass (Section 3.2 ). (c) Reusing the K/V computed during denoising as history allows multi-GPU stage pipelining and removes per-block recaching (Section 3.5 ).

Figure 6: Chunkwise Interactive qualitative comparison. Ours tracks the changing ten-second prompts with faithful semantics and stable visual quality, while competing methods exhibit quality degradation, object-count errors, or missed interactions.

Figure 7: Chunkwise MovieGen-100s qualitative comparison. Ours sustains the long-horizon table-wiping action with stable scene structure, whereas competing methods accumulate artifacts, duplicate subjects, or produce inconsistent reflections.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Two-pass training of SDF. (a) Pass 1 rolls out all blocks serially without gradients, recaches the sink from

\widehat{X}_{\Omega}^{(d)}

, and records the trajectory; the history of each block is read at the stage chosen by the random, fixed, or aligned policy. (b) Pass 2 replays the recorded states in one differentiable forward pass over a context stream

C

and a query stream

D

, so the DMD loss on the queries reaches the context K/V they attend to. Under the aligned policy (

c_{i}=d

), context and query coincide and the two streams collapse into one. For clarity, the attention masks omit the clean sink; as in SAF, block

i

also attends to the clean K/V of the preceding sink blocks,

\mathrm{KV}_{\Omega,<i}

(Section 3.4 ). At the top of (a),

X_{i}

is block

i

, whose K/V enters the history at the bold stage; the context and query streams in (b) hold its recorded states

X_{i}^{(c_{i})}

and

X_{i}^{(d)}

, and

c_{g}

is shared by context group

g

(Eq. 7 ).

Benchmark	Prompts	Samples	Length	Dimensions	Aggregate metrics
VBench	944	5	5 s	16	Total, Quality, Semantic
Interactive	$100\times 6$	1	60 s	7	Quality, ViCLIP
MovieGen-100s	128	1	100 s	7	Quality

Appendix

Table 5: Evaluation benchmarks. All models are trained on 5-second clips, so Interactive and MovieGen-100s test generation at

12{\times}

and

20{\times}

the training length. Each Interactive video is driven by six prompts in turn, each lasting 10 seconds, and the benchmark provides 100 such prompt sequences (

100\times 6

). Samples is the number of videos per prompt (per prompt sequence for Interactive), and Dimensions is the number of VBench dimensions scored.

Figure 9: Additional chunkwise Interactive comparison. Zoom in for better visualization.

Figure 10: Additional chunkwise MovieGen-100s example 1. Zoom in for better visualization.

Figure 11: Additional chunkwise MovieGen-100s example 2. Zoom in for better visualization.

Figure 12: Additional framewise Interactive comparison. Zoom in for better visualization.

Figure 13: Additional framewise MovieGen-100s example 1. Zoom in for better visualization.

Figure 14: Additional framewise MovieGen-100s example 2. Zoom in for better visualization.

Explore similar work

End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
Dec 17, 2025Yuwei Guo, Ceyuan Yang, Hao He +5Autoregressive Video Diffusion Models Resampling
The Chinese University of Hong Kong, Hong Kong, China · ByteDance Seed, China · ByteDance, China
FreqForcing: Autoregressive Long Video Generation via Spectral Self-Anchoring
Jul 29, 2026Jiatong Li, Leo Liang, Linghe Kong +1Autoregressive Video Diffusion Models Autoregressive Video Generation
Shanghai Jiao Tong University · Tencent HY Team
Stream Forcing: Constructing Unified Training Trajectory for Robust Streaming Video Generation
Aug 11, 2026Yueting Zhu, Yuehao Song, Kaicheng Zhang +5Video Diffusion Models Video Generation
Huazhong University of Science & Technology · Anyverse Dynamics · Horizon Robotics

Dec 17, 2025, cs.CVJ/K move · Enter open · S save

End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

Yuwei Guo, Ceyuan Yang, Hao He, Yang Zhao+4

The Chinese University of Hong Kong, Hong Kong, China · ByteDance Seed, China · ByteDance, China

Autoregressive Video Diffusion Models Resampling

Open paper AskPDF arXiv

Autoregressive video diffusion models hold promise for world simulation but are vulnerable to exposure bias arising from the train-test mismatch. While recent works address this via post-training, they typically rely on a bidirectional teacher model or discriminator. To achieve an end-to-end solution, we introduce Resampling Forcing, a teacher-free framework that enables training autoregressive video models from scratch and at scale. Central to our approach is a self-resampling scheme that simulates inference-time model errors on history frames during training. Conditioned on these degraded histories, a sparse causal mask enforces temporal causality while enabling parallel training with frame-level diffusion loss. To facilitate efficient long-horizon generation, we further introduce history routing, a parameter-free mechanism that dynamically retrieves the top-k most relevant history frames for each query. Experiments demonstrate that our approach achieves performance comparable to distillation-based baselines while exhibiting superior temporal consistency on longer videos owing to native-length training.

Abstract

Explore similar work

End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

FreqForcing: Autoregressive Long Video Generation via Spectral Self-Anchoring

Stream Forcing: Constructing Unified Training Trajectory for Robust Streaming Video Generation