Organizations: South China University of Technology · Westlake University · Johns Hopkins University · University of California, Merced · The Chinese University of Hong Kong
Large pretrained diffusion models have significantly enhanced the quality of generated videos, and yet their use in real-time streaming remains limited. Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity. Diffusion distillation can compress these models into efficient few-step variants, but existing video distillation approaches largely adapt image-specific methods that neglect temporal dependencies. These techniques often excel in image generation but underperform in video synthesis, exhibiting reduced motion coherence, error accumulation over long sequences, and a latency-quality trade-off. We identify two factors that result in these limitations: insufficient utilization of temporal context during step reduction and implicit prediction of subsequent noise levels in next-chunk prediction (i.e., exposure bias). To address these issues, we propose Diagonal Distillation, which operates orthogonally to existing approaches and better exploits temporal information across both video chunks and denoising steps. Central to our approach is an asymmetric generation strategy: more steps early, fewer steps later. This design allows later chunks to inherit rich appearance information from thoroughly processed early chunks, while using partially denoised chunks as conditional inputs for subsequent synthesis. By aligning the implicit prediction of subsequent noise levels during chunk generation with the actual inference conditions, our approach mitigates error propagation and reduces oversaturation in long-range sequences. We further incorporate implicit optical flow modeling to preserve motion quality under strict step constraints. Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.
Figures & tables
Figure 1: Diagonal Distillation achieves comparable quality to the full-step model while significantly reducing latency. The method yields a 1.88× speedup on 5-second short video generation on a single H100 GPU.
Figure 2: When the training data uses explicit noise frames as conditions in Causvid ( Yin et al., 2025 ) , the next chunk prediction essentially functions as an implicit next noise level prediction. We observe that even with single-step prediction, the image progressively becomes clearer.
Figure 3: Diagonal Denoising with Diagonal Forcing and Progressive Step Reduction. We give an illustration of our method by starting with five denoising steps for the first chunk and gradually reducing them to two steps by Chunk 7. For chunks with k≥4 , we use a fixed two-step denoising process, reusing the Key-Value (KV) cache from the final noisy frame of the preceding chunk. This design preserves temporal coherence while minimizing latency, and the corresponding pseudo-code is provided in the appendix.
Figure 4: Comparing the results from three different models. For more results, please refer to the Appendices.
Model
Throughput ↑
First-Frame Latency ↓
Speedup
Total ↑
Quality ↑
Semantic ↑
Wan2.1 ( Wan et al., 2025 )
0.78
103
1.0 ×
84.26
85.30
80.09
SkyReels-V2 ( Chen et al., 2025 )
0.49
112
0.91 ×
82.67
84.70
74.53
MAGI-1 ( Teng et al., 2025 )
0.19
282
0.36 ×
79.18
82.04
67.74
Causvid ( Yin et al., 2025 )
17.0
0.69
149.3 ×
81.20
84.05
69.80
Self-Forcing ( Huang et al., 2025 )
17.0
0.69
149.3 ×
84.31
85.07
81.28
DiagDistill (Ours)
31.0
0.37
277.3 ×
84.48
85.26
81.73
Table 1: Comprehensive comparison of video generation methods
Figure 5: Ablation study results. (a) Performance evaluation across different diagonal forcing timesteps, demonstrating optimal outcomes at 100 steps (1000 steps correspond to complete noise addition, while 0 steps represent the clean frame);(b)Impact of motion loss weight on model performance.
Figure 6: Visual comparison of motion effects. (a) Without motion loss shows minimal motion amplitude with only slight object movement; (b) With motion loss demonstrates significantly increased motion amplitude throughout the entire frame, validating our method’s effectiveness.
Figure 7: Qualitative comparison of long video generation (45s) with Self-Forcing and Causvid. The visual results show that other methods suffer from noticeable saturation distortion and quality decay over time, whereas our approach preserves detail and consistency. Additional results are provided in the supplementary material.
Ablation Variant Temporal Quality ↑ Frame Quality ↑ Text Alignment ↑ Total Score ↑ Without Diagonal Forcing 92.1 60.1 26.9 83.58 Without Flow Loss 92.5 60.8 27.8 84.18 Without Diagonal Denoising 95.1 63.2 28.6 84.46 Full Method (Ours) 94.9 63.4 28.9 84.48 Table 2: Ablation Study on Key Components of DiagDistill.
Table 2: Ablation Study on Key Components of DiagDistill.
Figure 8: Quantitative evaluation of long video generation. The plot compares human preference scores and quality consistency over time for different methods under identical conditions. The results show that our approach maintains a stable quality throughout extended sequences, achieving scores above 50%, and attains a significant reduction in inference latency.
Figure 9: Illustration of long video generation with dynamic prompting. This feature allows for the integration of new prompts at arbitrary time points, facilitating the creation of coherent long videos with changing narratives. The specific prompts used for each segment are detailed in the appendix.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Comparative visualization of temporal training strategies for autoregressive video generation using Causal DiT. Four panels illustrate: (a) Teacher Forcing (green boxes for ground-truth frames), (b) Diffusion Forcing (red boxes for noisy latents), (c) Self Forcing (red boxes for model’s own predictions), and (d) Diagonal Forcing (Ours) (mixed green/red boxes in diagonal patterns). Each row represents sequential frame generation, with arrows indicating causal dependencies. The diagonal pattern in (d) highlights the core innovation—blending clean past frames with recent model-generated ones to align training/inference distributions. This visual comparison underscores how Diagonal Forcing bridges gaps in robustness and coherence seen in baseline methods.
Attention Window Size
Total Score
In-Flight Latency (s)
Memory (GB)
3
80.9
0.37±0.01
14.9
6
81.3
0.38±0.01
15.8
9
82.3
0.40±0.01
16.6
12
84.3
0.46±0.02
17.5
15
84.2
0.51±0.02
18.4
18
84.4
0.54±0.02
19.2
Appendix
Table 4: KV cache scaling analysis
Total NFEs
Mode
Steps Allocation
Temporal Quality ↑
Frame Quality ↑
Text Alignment ↑
In-Flight Latency (s) ↓
Throughput (FPS) ↑
34
Diagonal
4322222
94.9
63.4
28.9
0.23±0.02
31.0
34
Self
4322222
93.5
62.1
27.5
0.43±0.02
25.9
40
Diagonal
5432222
94.8
63.1
29.0
0.23±0.02
29.7
40
Self
5432222
93.9
62.3
28.1
0.43±0.02
23.8
44
Diagonal
4333333
95.0
63.7
28.5
0.34±0.02
23.5
44
Self
4333333
94.2
62.8
27.8
0.56±0.02
21.5
Appendix
Table 5: Comparative evaluation under identical NFE budgets demonstrates Diagonal Forcing’s superior efficiency. Our method achieves better quality metrics with significantly lower latency and higher throughput across all configurations.
Schedule
Temporal Quality ↑
Frame Quality ↑
Text Alignment ↑
NFEs
In-Flight Latency (s) ↓
Throughput (FPS) ↑
Baseline Strategies (From Paper)
5333333
95.0
63.9
29.1
46
0.34±0.02
22.5
4333333
95.0
63.7
28.5
44
0.34±0.02
23.5
5433333
95.1
63.2
29.3
48
0.34±0.02
23.3
5432222
94.8
63.1
29.0
40
0.23±0.02
29.7
4322222
94.9
63.4
28.9
34
0.23±0.02
31.0
Appendix
Table 6: Quantitative comparison of different denoising step allocation strategies. Strategies are grouped by type: Baseline Strategies (evaluated in the main paper), Exploration: Emphasizing Early Frames, and Exploration: Non-monotonic / Dynamic Allocation. NFEs denotes the total number of function evaluations. Best results are in bold.
Type
Throughput (FPS) ↑
Latency (s) ↓
Speedup ↑
Total Score
Quality
Semantics
DiagDistill (Chunk-level, chunk size=3)
31.0
0.37
277.3x
84.48
85.26
81.73
DiagDistill (Frame-level, chunk size=1)
16.5
0.25
412.0x
84.29
85.41
80.40
Appendix
Table 7: Performance comparison of chunk-level (size=3) vs. frame-level (size=1) streaming variants
Model
Decoder Params
Compression Rate
Decoding Time(s) (81 × 832 × 480)
Full VAE (Wan 2.1)
73.3M
8 × 8 × 4
1.67
Tiny VAE (Wan 2.1) (Boer Bohan, 2025)
9.84M
8 × 8 × 4
0.12
Appendix
Table 8: Comparison of Causal VAE Models.
Flow Representation Variant
Temporal Quality ↑
Frame Quality ↑
Text Alignment ↑
Raw Latent
92.5
60.8
27.8
Latent Difference
93.8
61.5
28.2
Latent Correlation
94.2
62.1
28.4
Low Frequency Components
93.5
61.8
28.1
High Frequency Components
94.0
62.3
28.5
Learnable MLP
94.6
63.2
28.7
Appendix
Table 9: Performance comparison of different F(⋅) implementations.