Organizations: South China University of Technology · Westlake University · Johns Hopkins University · University of California, Merced · The Chinese University of Hong Kong
Large pretrained diffusion models have significantly enhanced the quality of generated videos, and yet their use in real-time streaming remains limited. Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity. Diffusion distillation can compress these models into efficient few-step variants, but existing video distillation approaches largely adapt image-specific methods that neglect temporal dependencies. These techniques often excel in image generation but underperform in video synthesis, exhibiting reduced motion coherence, error accumulation over long sequences, and a latency-quality trade-off. We identify two factors that result in these limitations: insufficient utilization of temporal context during step reduction and implicit prediction of subsequent noise levels in next-chunk prediction (i.e., exposure bias). To address these issues, we propose Diagonal Distillation, which operates orthogonally to existing approaches and better exploits temporal information across both video chunks and denoising steps. Central to our approach is an asymmetric generation strategy: more steps early, fewer steps later. This design allows later chunks to inherit rich appearance information from thoroughly processed early chunks, while using partially denoised chunks as conditional inputs for subsequent synthesis. By aligning the implicit prediction of subsequent noise levels during chunk generation with the actual inference conditions, our approach mitigates error propagation and reduces oversaturation in long-range sequences. We further incorporate implicit optical flow modeling to preserve motion quality under strict step constraints. Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.
Figures & tables
Figure 1: Diagonal Distillation achieves comparable quality to the full-step model while significantly reducing latency. The method yields a 1.88× speedup on 5-second short video generation on a single H100 GPU.
Figure 2: When the training data uses explicit noise frames as conditions in Causvid ( Yin et al., 2025 ) , the next chunk prediction essentially functions as an implicit next noise level prediction. We observe that even with single-step prediction, the image progressively becomes clearer.
Figure 3: Diagonal Denoising with Diagonal Forcing and Progressive Step Reduction. We give an illustration of our method by starting with five denoising steps for the first chunk and gradually reducing them to two steps by Chunk 7. For chunks with k≥4 , we use a fixed two-step denoising process, reusing the Key-Value (KV) cache from the final noisy frame of the preceding chunk. This design preserves temporal coherence while minimizing latency, and the corresponding pseudo-code is provided in the appendix.
Figure 4: Comparing the results from three different models. For more results, please refer to the Appendices.
Model
Throughput ↑
First-Frame Latency ↓
Speedup
Total ↑
Quality ↑
Semantic ↑
Wan2.1 ( Wan et al., 2025 )
0.78
103
1.0 ×
84.26
85.30
80.09
SkyReels-V2 ( Chen et al., 2025 )
0.49
112
0.91 ×
82.67
84.70
74.53
MAGI-1 ( Teng et al., 2025 )
0.19
282
0.36 ×
79.18
82.04
67.74
Causvid ( Yin et al., 2025 )
17.0
0.69
149.3 ×
81.20
84.05
69.80
Self-Forcing ( Huang et al., 2025 )
17.0
0.69
149.3 ×
84.31
85.07
81.28
DiagDistill (Ours)
31.0
0.37
277.3 ×
84.48
85.26
81.73
Table 1: Comprehensive comparison of video generation methods
Figure 5: Ablation study results. (a) Performance evaluation across different diagonal forcing timesteps, demonstrating optimal outcomes at 100 steps (1000 steps correspond to complete noise addition, while 0 steps represent the clean frame);(b)Impact of motion loss weight on model performance.
Figure 6: Visual comparison of motion effects. (a) Without motion loss shows minimal motion amplitude with only slight object movement; (b) With motion loss demonstrates significantly increased motion amplitude throughout the entire frame, validating our method’s effectiveness.
Figure 7: Qualitative comparison of long video generation (45s) with Self-Forcing and Causvid. The visual results show that other methods suffer from noticeable saturation distortion and quality decay over time, whereas our approach preserves detail and consistency. Additional results are provided in the supplementary material.
Ablation Variant Temporal Quality ↑ Frame Quality ↑ Text Alignment ↑ Total Score ↑ Without Diagonal Forcing 92.1 60.1 26.9 83.58 Without Flow Loss 92.5 60.8 27.8 84.18 Without Diagonal Denoising 95.1 63.2 28.6 84.46 Full Method (Ours) 94.9 63.4 28.9 84.48 Table 2: Ablation Study on Key Components of DiagDistill.
Table 2: Ablation Study on Key Components of DiagDistill.
Figure 8: Quantitative evaluation of long video generation. The plot compares human preference scores and quality consistency over time for different methods under identical conditions. The results show that our approach maintains a stable quality throughout extended sequences, achieving scores above 50%, and attains a significant reduction in inference latency.
Figure 9: Illustration of long video generation with dynamic prompting. This feature allows for the integration of new prompts at arbitrary time points, facilitating the creation of coherent long videos with changing narratives. The specific prompts used for each segment are detailed in the appendix.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Comparative visualization of temporal training strategies for autoregressive video generation using Causal DiT. Four panels illustrate: (a) Teacher Forcing (green boxes for ground-truth frames), (b) Diffusion Forcing (red boxes for noisy latents), (c) Self Forcing (red boxes for model’s own predictions), and (d) Diagonal Forcing (Ours) (mixed green/red boxes in diagonal patterns). Each row represents sequential frame generation, with arrows indicating causal dependencies. The diagonal pattern in (d) highlights the core innovation—blending clean past frames with recent model-generated ones to align training/inference distributions. This visual comparison underscores how Diagonal Forcing bridges gaps in robustness and coherence seen in baseline methods.
Attention Window Size
Total Score
In-Flight Latency (s)
Memory (GB)
3
80.9
0.37±0.01
14.9
6
81.3
0.38±0.01
15.8
9
82.3
0.40±0.01
16.6
12
84.3
0.46±0.02
17.5
15
84.2
0.51±0.02
18.4
18
84.4
0.54±0.02
19.2
Appendix
Table 4: KV cache scaling analysis
Total NFEs
Mode
Steps Allocation
Temporal Quality ↑
Frame Quality ↑
Text Alignment ↑
In-Flight Latency (s) ↓
Throughput (FPS) ↑
34
Diagonal
4322222
94.9
63.4
28.9
0.23±0.02
31.0
34
Self
4322222
93.5
62.1
27.5
0.43±0.02
25.9
40
Diagonal
5432222
94.8
63.1
29.0
0.23±0.02
29.7
40
Self
5432222
93.9
62.3
28.1
0.43±0.02
23.8
44
Diagonal
4333333
95.0
63.7
28.5
0.34±0.02
23.5
44
Self
4333333
94.2
62.8
27.8
0.56±0.02
21.5
Appendix
Table 5: Comparative evaluation under identical NFE budgets demonstrates Diagonal Forcing’s superior efficiency. Our method achieves better quality metrics with significantly lower latency and higher throughput across all configurations.
Schedule
Temporal Quality ↑
Frame Quality ↑
Text Alignment ↑
NFEs
In-Flight Latency (s) ↓
Throughput (FPS) ↑
Baseline Strategies (From Paper)
5333333
95.0
63.9
29.1
46
0.34±0.02
22.5
4333333
95.0
63.7
28.5
44
0.34±0.02
23.5
5433333
95.1
63.2
29.3
48
0.34±0.02
23.3
5432222
94.8
63.1
29.0
40
0.23±0.02
29.7
4322222
94.9
63.4
28.9
34
0.23±0.02
31.0
Appendix
Table 6: Quantitative comparison of different denoising step allocation strategies. Strategies are grouped by type: Baseline Strategies (evaluated in the main paper), Exploration: Emphasizing Early Frames, and Exploration: Non-monotonic / Dynamic Allocation. NFEs denotes the total number of function evaluations. Best results are in bold.
Type
Throughput (FPS) ↑
Latency (s) ↓
Speedup ↑
Total Score
Quality
Semantics
DiagDistill (Chunk-level, chunk size=3)
31.0
0.37
277.3x
84.48
85.26
81.73
DiagDistill (Frame-level, chunk size=1)
16.5
0.25
412.0x
84.29
85.41
80.40
Appendix
Table 7: Performance comparison of chunk-level (size=3) vs. frame-level (size=1) streaming variants
Model
Decoder Params
Compression Rate
Decoding Time(s) (81 × 832 × 480)
Full VAE (Wan 2.1)
73.3M
8 × 8 × 4
1.67
Tiny VAE (Wan 2.1) (Boer Bohan, 2025)
9.84M
8 × 8 × 4
0.12
Appendix
Table 8: Comparison of Causal VAE Models.
Flow Representation Variant
Temporal Quality ↑
Frame Quality ↑
Text Alignment ↑
Raw Latent
92.5
60.8
27.8
Latent Difference
93.8
61.5
28.2
Latent Correlation
94.2
62.1
28.4
Low Frequency Components
93.5
61.8
28.1
High Frequency Components
94.0
62.3
28.5
Learnable MLP
94.6
63.2
28.7
Appendix
Table 9: Performance comparison of different F(⋅) implementations.
Video generation has recently emerged as a central task in the field of generative AI. However, the substantial computational cost inherent in video synthesis makes model distillation a critical technique for efficient deployment. Despite its significance, there is a scarcity of methods specifically designed for video diffusion models. Prevailing approaches often directly adapt image distillation techniques, which frequently lead to artifacts such as oversaturation, temporal inconsistency, and mode collapse. To address these challenges, we propose a novel distillation framework tailored specifically for video diffusion models. Its core innovations include: (1) an adaptive regression loss that dynamically adjusts spatial supervision weights to prevent artifacts arising from excessive distribution shifts; (2) a temporal regularization loss to counteract temporal collapse, promoting smooth and physically plausible sampling trajectories; and (3) an inference-time frame interpolation strategy that reduces sampling overhead while preserving perceptual quality. Extensive experiments and ablation studies on the VBench and VBench2 benchmarks demonstrate that our method achieves stable few-step video synthesis, significantly enhancing perceptual fidelity and motion realism. It consistently outperforms existing distillation baselines across multiple metrics.
We propose OPSD-V, an on-policy self-distillation paradigm for post-training few-step autoregressive (AR) video diffusion models. Existing few-step AR video generators can produce long videos with low latency, but still suffer from error accumulation and weakened motion dynamics during long autoregressive rollout. OPSD-V reduces long-horizon degradation while preserving the original few-step inference path. The key idea is to introduce real long-video data as temporal context during training and use it to provide dense trajectory-level supervision. Specifically, the student follows the exact inference-time rollout, generating each chunk conditioned on its own previously generated KV cache. In parallel, the teacher is evaluated at the same student-visited denoising states, but uses a cleaner AR-consistent temporal cache in which older history can be replaced by real-video context. This provides dense denoising-level corrective targets under on-policy AR cache dynamics, without changing the sampler, number of denoising steps, or inference-time cache mechanism. We apply OPSD-V to representative few-step AR video models, including Self-Forcing and LongLive. Experiments show consistent improvements in visual quality, motion dynamics, and VBenchLong scores. A user study with 10 participants comparing 20 video pairs shows that OPSD-V is preferred over the base models in 66.0% of overall-preference judgments (82.5% excluding ties).
Autoregressive video diffusion models have become a leading approach to real-time interactive video generation and video world modeling. However, existing methods typically allocate computation uniformly across space when generating the next video chunk, updating all patches with the same denoising schedule and number of function evaluations. This ignores the spatial heterogeneity of video content: static, simple, or recurring regions are often easier to denoise, while other regions require further refinement or additional context. A uniform budget thus leads to redundant computation on easy patches, whereas reducing the fixed step count can leave difficult patches insufficiently refined. We introduce \methodname{}, a general framework that dynamically allocates denoising steps to individual patches: easy patches stop early to improve efficiency, while difficult patches receive more refinement steps. This design accelerates generation with negligible quality degradation. Our method is compatible with most existing few-step autoregressive diffusion models, requiring only a lightweight routing module with modest training. Experiments demonstrate that \methodname{} reduces the total number of denoising steps applied across video patches by about 50% while maintaining visual quality comparable to the four-step baseline.
Zekai Gu, Peng Wang, Jiahao Lu +3
HKUST · VAST · Macau University of Science and Technology