Autoregressive video diffusion models (AR-VDMs) enable real-time, long-horizon video generation but degrade rapidly once the inference horizon outruns the training horizon, due to compounding exposure bias. To stabilize long rollouts, recent methods heuristically adopt frame sink, keeping initial frames as a persistent model context, inspired by attention sinks in LLMs. We revisit this design and find that the mechanism by which initial frames stabilize long-range consistency in AR-VDMs is not an LLM-style attention sink; instead, long-range consistency is anchored through a sparse subset of attention heads, while the majority of heads predominantly capture local temporal dynamics. This observation reveals an over-conditioning pathology in existing frame-sink strategies: by universally exposing every head to the pinned initial-frame context, the model becomes prone to collapsing motion toward the low-dynamic modes of the teacher during model distillation. We propose Disen-Forcing, a distillation-time plug-in for AR-VDMs that disentangles and reinforces this head specialization through disentangled context routing and motion-disentangling regularization. On long-horizon video generation benchmarks, Disen-Forcing simultaneously improves long-range consistency and motion dynamics of base AR-VDMs, a trade-off that sink-based methods fail to balance.
Figures & tables
Figure 2: Temporal attention patterns across models and heads. (a) Per-head attention to the initial and current frames across AR-VDMs, averaged over 50 MovieGenBench [ 32 ] prompts and denoising timesteps. Most heads attend predominantly to the current frame, while a sparse subset concentrates attention on the initial frame. (b) Attention distributions of representative Self-Forcing heads across seven rollout steps, at ODE initialization (left) and after DMD (right). Working Heads exhibit local temporal patterns, whereas Anchor Heads attend persistently to the initial frame.
Figure 3: Critical Role of Anchor Heads in Semantic Consistency. Masking the top 20% Anchor Heads leads to a sharp drop in subject/background consistency and a clear rise in imaging-quality drift (left), together with visible appearance drift even within the 5-second training horizon (right). Masking the same number of Working Heads at random yields metrics on par with the unmasked baseline and no observable drift.
Figure 4: Anchor-head positions are input invariant. (a) Each row plots a single prompt’s per-head initial-frame attention score. The same vertical stripes are coherent across all 50 rows, meaning the same subset of heads attends strongly to the initial frame regardless of prompt. (b) Each row plots the per-head cross-prompt mean attention score at one of the four diffusion timesteps. The four rows display the same vertical stripes, indicating timestep-invariance as well.
Figure 5: Disentangled context routing. (a) Sliding window attention loses early frames as the cache rolls forward. (b) Sinked sliding window retains initial frames but routes the same context to all heads. (c) During DMD, our approach routes initial-frame context exclusively to the sparse Anchor Heads while the majority Working Heads attend only to the local sliding window, preserving the natural head specialization.
Figure 6: Qualitative comparison and ablation. Self-Forcing suffers from temporal drift once the inference rollout exceeds the training horizon; Frame Sink avoids temporal drift but yields low-dynamic, conservative motion. Ours (Disen-Forcing) is both drift-free and temporally dynamic. The bottom block isolates the contributions of our two components: context routing and Lreg .
VBench-Long Scores ↑
ΔDriftQ↓
Model
Total
Subject
Background
Smooth
Dynamic
Aesthetic
Imaging
30-second generation ( ∼ 500 frames)
Diffusion Forcing
SkyReels-V2 [ 5 ]
76.64
96.89
96.98
99.14
45.50
60.04
61.31
5.65
Distilled Causal
CausVid [ 52 ]
78.44
97.83
96.72
98.15
52.59
59.33
66.01
6.73
Table 1: Quantitative comparison on long video generation. Disen-Forcing is a distillation-time plug-in applied to Self-Forcing and Causal-Forcing. ΔDriftQ ( ↓ ) measures the imaging quality gap between the first and last segments. Best Total per horizon in bold .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Computation flow of the motion-disentangling regularization. The standard and counterfactual passes share the same input and model parameters but use different context caches. Adjacent-frame attention maps from the two passes are aligned by KL divergence, with the counterfactual map serving as a stop-gradient target.
Figure A2 : Representative per-(layer, head) attention map grid. Each cell shows the chunk-by-chunk cross-frame attention pattern of one (layer, head) pair throughout a 7-step AR rollout, averaged over 50 MovieGenBench prompts and all diffusion timesteps. Red -bordered cells are Anchor Heads (top-20% by initial-frame attention score, p=80 ); blue -bordered cells are Working Heads . The eight heads visualized in Fig. 2(b) are sampled from this grid; the representative visualization confirms that the anchor/working partition is sparse, layer-localized (notably layers 18 and 22–24), and consistent across both base models. Zoom in for details.
Figure A3 : Per-head attention distribution for additional baselines. Same head selection and rendering as Fig. 2(b) , applied to Causal-Forcing (sliding-window) and the two sink-based methods LongLive and Rolling-Forcing. The same anchor/working partition appears in all three distillation pipelines.
Figure A4 : IoU of the p80 -thresholded Anchor Heads set AK identified from K random prompts versus the reference A103 . Set size ∣AK∣=72 at every K , out of 360 total attention heads.
Method
Critic (s)
Generator (s)
Gen:critic
Amortized (s)
+ Frame Sink
7.0
16.1
1:5
10.2
+Ours
7.0
18.8
1:5
10.8 ( +5% )
Appendix
Table A1 : Training overhead. Critic and generator columns report mean forward-and-backward time per pass. The amortized time is (5tcritic+tgen)/5 under the 1:5 generator-to-critic update schedule.
Figure A5 : VideoScore preference rates on 30s generations. Percentages compare the three methods within each base model and evaluation dimension.
Figure A6 : Qualitative anchor-head masking comparison across prompts. Each row shows one prompt rendered under three masking conditions: no mask (baseline, left block), mask random heads (control, middle block, blue ), and mask anchor heads (treatment, right block, red ). Each block displays three frames spanning the 5-second clip. A brief prompt description is shown below each row. Random masking preserves visual fidelity; anchor masking produces visible identity, color, and scene drift, confirming that long-range anti-drift is carried specifically by the anchor-head subset.
Figure A7 : 1-minute qualitative comparison.
Figure A8 : 1-minute qualitative comparison.
Figure A9 : 30-second qualitative comparison.
Figure A10 : 30-second qualitative comparison.
Figure A11 : Per-head temporal attention patterns in bidirectional Wan base models. Attention scores averaged over 50 prompts and diffusion timesteps for the pretrained bidirectional Wan2.1-1.3B (left, 30 layers, 12 heads), Wan2.1-14B (middle, 40 layers, 40 heads), and Wan2.2-TI2V-5B (right, 30 layers, 24 heads), before any causal AR training or distillation. All three models exhibit the same structural pattern observed in the AR-VDM variants of Fig. 2(a) : most heads attend predominantly to the current frame, while a sparse subset at specific layers shows concentrated attention to the initial frame.
Figure A12 : Reconstruction-based motion-proxy layer analysis. Left: reconstruction error for candidate layers 20, 26, 27, and 28 from warping the reference frame along cross-frame attention chains, normalized to a no-motion baseline (mean ± std over motion prompts × seeds); layer 26 has the lowest error. Right: example attention-based point tracks on a running dog (top) and a leaping deer (bottom).
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
Autoregressive video diffusion models support real-time synthesis but suffer from error accumulation and context loss over long horizons. We discover that attention heads in AR video diffusion transformers serve functionally distinct roles as local heads for detail refinement, anchor heads for structural stabilization, and memory heads for long-range context aggregation, yet existing methods treat them uniformly, leading to suboptimal KV cache allocation. We propose Head Forcing, a training-free framework that assigns each head type a tailored KV cache strategy: local and anchor heads retain only essential tokens, while memory heads employ a hierarchical memory system with dynamic episodic updates for long-range consistency. A head-wise RoPE re-encoding scheme further ensures positional encodings remain within the pretrained range. Without additional training, Head Forcing extends generation from 5 seconds to minute-level duration, supports multi-prompt interactive synthesis, and consistently outperforms existing baselines. Project Page: https://jiahaotian-sjtu.github.io/headforcing.github.io/.
Jiahao Tian, Yiwei Wang, Gang Yu +1
AGI Lab, Westlake University · University of California at Merced · StepFun
Autoregressive (AR) video diffusion enables low-latency, streamable video generation, but prediction errors often accumulate over long rollouts. Training the generator on its own rollouts exposes it to these imperfect histories. However, existing video-level distribution matching distillation (DMD) scores the whole rollout jointly. Because a chunk is evaluated together with its past and future, its correction can favor matching artifacts in the surrounding context merely to preserve temporal consistency. To provide a clearer visual-quality signal, we introduce Rollout-Marginal Distillation (RMD). RMD retains the generated history for AR prediction but scores each chunk independently against a chunk teacher, ensuring its quality correction is not compromised by an imperfect temporal context. To compensate for the lack of temporal context in independent chunk scoring, RMD subsequently applies video-level DMD to restore temporal coherence. Extensive experiments demonstrate that RMD maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines. Code and video results are available at https://cjeen.github.io/RMD
Chenjian Gao, Zhihao Hu, Jianqi Ma +3
MMLab, The Chinese University of Hong Kong · Tencent AIPD
Autoregressive (AR) modeling is arguably the only way to extend a fixed-length video diffusion model to longer or even streaming videos. However, existing AR diffusion methods use only the previously predicted chunk as context for the next, discarding the intermediate predictions that constitute the chunk's generation history. In contrast, conventional AR models such as LLMs condition on the entire history of preceding tokens. To this end, we propose Trajectory Forcing (TraF), an AR diffusion method that exploits the denoising trajectories of previous video chunks as context. TraF operates under two training regimes: (1) with ground-truth videos, where pseudo-trajectories are constructed from Gaussian interpolations and re-predicted by the model, and (2) with a teacher model, where real trajectories are obtained from the student's own denoising rollout. The first regime naturally provides a strong initialization for the second. On the 5-second VBench benchmark, TraF achieves a Total score of 85.15, outperforming Self Forcing (+0.84) and Causal Forcing (+1.11), with Quality and Semantic improving simultaneously. On 30-second generation---6× the training horizon---TraF achieves the highest overall quality among all compared methods.