Organizations: DENG Lab, Shanghai Jiao Tong University · Xi’an Jiaotong University · Peking University · East China Normal University · National Tsing Hua University
Real-time long-video generation requires continuous output with consistent appearance and natural motion. Existing chunk-wise autoregressive methods face challenges in both generation efficiency and long-video quality: serialized denoising and history-cache refresh require two forward passes even with one-step denoising, and longer rollouts can degrade quality and cause camera drift. We propose Wave Forcing (WF), which co-designs generation, training, and execution for continuous video output with multi-step denoising. WF uses chunk-wise causal attention to access earlier, cleaner chunks before their denoising is complete, and distributes staggered denoising and cache-refresh stages across full-model replicas to achieve 1F1C: one output chunk per concurrent forward round after pipeline fill. To adapt the model to this continuous generation process, we develop a three-stage training procedure that successively introduces mixed-noise denoising, motion supervision, and causal adaptation using self-generated history. To execute this process efficiently, Wave Runtime (WaveRT) uses Wave Parallelism (WP) for early K/V delivery and communication-computation overlap, together with optimized kernels and pipelined VAE decoding. Experiments on Wan2.1-T2V-1.3B show that 5-step WF maintains overall quality comparable to Rolling Forcing while achieving higher Dynamic Degree on both short and long videos. Our system efficiency experiments show that 4-step WF achieves 147.6 DiT FPS and 134.8 end-to-end FPS on eight H200 GPUs, yielding 7.07x the single-GPU end-to-end throughput. In DiT-only tests with matched SageAttention+W8A8 backends, denoising steps, and GPU counts, WP delivers up to 2.53x the throughput of the faster of the Ulysses and Ring context-parallel baselines across six model scales (1.3B-14B).
Figures & tables
Figure 1: Causal wavefront and continuous output ( S=4 ). (a) Denoising and store advance on anti-diagonals, with dashed cells marking the next round. (b) One later-layer K/V edge arrives before use. Other edges and initial waits are omitted. (c) Both schedules retain four denoising forwards plus store per chunk. Over five rounds, one SF replica emits one chunk, whereas five WF replicas emit five after fill. Wall-clock gains depend on stage costs and overlap.
Figure 2: Training flow. Stage 1 applies DMD to one sampled RF or SF rollout. Stage 2 re-noises teacher chunks and applies DMD and LPIPS to the same prediction Yθ . Stage 3 samples one self-generated RF, WF, or SF rollout for on-policy DMD.
Figure 3: Hiding K/V delivery with identical schematic operation costs. (a) Layer barriers expose communication. (b) Early publication hides later waits, not dependencies or the initial wait. (c) Direct writes to receiver slots precede readiness signals and consumption.
Figure 4: Streaming VAE schedules. Panels (a,b) compare blocking and asynchronous decoding with matched costs. Panels (c,d) compare unbalanced and load-balanced PP under identical workloads and arrivals, with darker teal indicating longer stage service times. Panel (e) shows profile-guided contiguous decoder partitions. Each comparison pair uses its own schematic time scale, with transfers omitted and partition boundaries shown schematically. The first output incurs pipeline fill latency (Appendix B.5 ) .
Standard VBench ↑
VBench-Long ↑
Stability
FPS ↑
Method
Steps
GPUs
Total
Quality
Semantic
Dynamic
Quality
Dynamic
Subj. ↑
Drift ↓
DiT
E2E
Rolling Forcing
5
1
81.15
83.77
70.66
51.39
81.18
36.63
92.75
0.17
29.12
17.46
+ Causal Attention
5
8
80.89
83.35
71.05
37.50
80.50
46.60
91.26
0.40
104.85
96.56
Causal Forcing (Full Attn.)
4
1
79.61
82.83
66.72
100.00
82.15
96.61
71.11
11.14
33.88
19.06
Wave Forcing (Ours)
4
8
80.74
83.93
67.95
84.72
80.89
74.97
91.88
0.36
147.58
134.81
+ SageAttention
8
81.03
84.02
69.07
65.28
81.16
77.24
90.79
0.36
147.78
134.79
Table 1: Short- and long-video comparison. Dynamic is Dynamic Degree. Subj./Drift are Subject Preservation/Camera Drift (scores: 0–100). The two FPS columns report DiT-only and end-to-end (E2E) runs.
Figure 5: Qualitative comparison. Rows show Causal Forcing, Rolling Forcing, and 4-step and 5-step Wave Forcing.
Model
Quality ↑
Dynamic ↑
Subj. ↑
Drift ↓
w/o off-policy DMD
76.68
15.73
76.53
0.26
w/o RF
80.19
88.82
77.48
6.24
w/o WF
80.34
48.12
95.64
0.10
w/o SF
75.35
23.66
84.04
1.89
Ours full
80.89
74.97
91.88
0.36
Table 2: 4-step training ablations on long videos. All rows evaluate final models. Quality/Dynamic are VBench-Long scores.
Figure 6: Qualitative training ablations. We compare the full 4-step model with each training ablation.
Figure 7: System efficiency with dummy DiT weights. (a) E2E throughput of 4-step WaveRT (8 GPUs) and the single-GPU baseline, with separate axes for 1.3B and 14B. (b) Steady-state tick p50 on the same WF graph, normalized to Sync. (c,d) BF16 DiT throughput across scales and WP’s percentage gain over the faster CP baseline. (e,f) DiT throughput across compute backends at 1.3B and 14B. Panels (c–f) disable VAE and use 5 steps/6 GPUs for 1.3B–9B and 4 steps/5 GPUs for 14B. Full measurements and repetition details appear in Tabs. 4 and 6 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Nchunk,i
Nchunk is the number of requested chunks, and i indexes chunks with 0≤i<Nchunk .
S,s,νs
S is the number of denoising updates, s indexes denoising stages with 0≤s<S , and νs is the diffusion timestep. Store occupies execution stage s=S .
k,r
k indexes forward rounds, and r indexes diffusion ranks with 0≤r≤S , where rank S is Store.
Zi(s),Bi
Zi(s) is the state after s updates, and Bi=Zi(S) denotes the clean endpoint.
Ak,Fk
active denoising-stage indices and their input states
Wk
complete wavefront state, including the clean input to store when active
Appendix
Table 3: Graph and runtime notation. The complete wavefront includes store, whereas the denoising subset does not .
Table 4: Runtime throughput. WaveRT uses the WF graph and paged K/V unless marked Sync, while Baseline is single-GPU rolling inference. Both use BF16 with FA3 unless a kernel suffix is specified. GPU counts include denoising, Store, and VAE: 4+1+3 for 4-step and 5+1+2 for 5-step. Speedup is E2E relative to the same-scale, same-step baseline. All rows use dummy DiT weights and a real-weight VAE. Best values are determined within each model/step group using unrounded measurements. DiT/E2E are paired .
Backbone
Compute
Sync
Overlap
One-sided
Staggered
Paged
1.3B
BF16
91.956
85.730
81.669
81.953
78.882
1.3B
Sage
91.441
84.103
80.934
81.062
78.356
1.3B
Sage+W8A8
88.714
81.595
78.030
78.081
76.052
14B
BF16
464.270
449.689
425.907
425.188
413.795
14B
Sage
467.526
439.988
426.215
425.601
414.029
14B
Sage+W8A8
407.215
379.576
365.922
366.697
354.447
Appendix
Table 5: WaveRT same-graph system ablations report pooled steady tick p50 (ms, lower is better) for the 4-step, 8-GPU (4+1+3) setup across three fixed repeats. Each row compares execution strategies under a fixed compute backend. All runs use dummy DiT weights and a real-weight VAE and measure execution cost only .
DiT throughput (FPS) ↑
Scale
Compute
Steps
GPUs
Ulysses CP
Ring CP
WP
WP/CP
1.3B
BF16
5
6
80.60
41.19
150.09 †
1.862× †
1.3B
Sage
5
6
84.01
56.99
151.15
1.799×
1.3B
Sage+W8A8
5
6
61.17
45.96
154.75
2.530×
3B
BF16
5
6
75.38
38.20
88.86
1.179×
3B
Sage
5
6
75.93
37.00
89.26
1.176×
Appendix
Table 6: DiT throughput across parallel strategies and compute backends, using dummy DiT weights with VAE disabled. Step counts and GPU counts are matched across methods within each row. FPS is the arithmetic mean of per-run rates, and best values are determined using unrounded measurements. WP/CP divides WP throughput by that of the faster CP baseline. Amber {\color[rgb]{0.582,0.4219,0.207}\dagger} marks supplemental measurements whose repetition and stopping protocols are described in Appendix D.2 .
Scale
Compute
Sync
Overlap
One-sided
Staggered
Paged
1.3B
BF16
91.946–92.009
85.659–85.763
81.589–81.744
81.842–82.107
78.851–78.967
1.3B
Sage
91.390–91.462
84.035–84.168
80.933–80.953
80.806–81.235
78.290–78.400
1.3B
Sage+W8A8
88.579–88.781
81.505–81.634
77.975–78.099
77.895–78.291
75.769–76.284
14B
BF16
463.785–464.761
449.001–450.082
425.517–426.670
424.276–425.844
413.094–414.374
14B
Sage
466.946–468.320
439.004–440.453
425.819–426.635
425.420–425.753
413.711–414.304
14B
Sage+W8A8
407.121–407.384
379.268–380.479
365.482–366.516
366.552–367.034
354.189–354.720
Appendix
Table 7: Minimum–maximum of per-run tick p50 (ms) across three fixed repetitions on the same node for causal 4-step, 8-GPU WaveRT with dummy DiT weights. These ranges are not confidence intervals.
Diagnostic
Measurement
Value
14B critical consumer
r0 forward interval
414.23 ms
14B producers
r1/r2/r3/store forward interval
370.50–379.37 ms
Aggregate rank-time difference
Critical minus mean of other ranks
38.32 ms
BF16 K/V communication
Rank 0 collective time summed over layers
58.45 ms
FP8 K/V communication
Payload and scale transfer, excluding conversion
31.23 ms
Copy-engine P2P
95,846,400-byte IPC copy
391.49 GB/s
Appendix
Table 8: Diagnostic-only component profiles of stage service, K/V transfer, receiver-side assembly, and VAE partitioning, using three-run means. Each row reports a component timing within its stated measurement scope. Because these components can overlap during execution, their durations cannot be summed to obtain E2E latency. The producer range spans rankwise means.
Schedule
Context state
Across-stage transfer
Execution structure
Self Forcing
Completed clean-history K/V
None
Sequential chunks
Live Avatar
Fixed-timestep stage-local K/V snapshots
Updated latent
Stage pipeline
HiAR
Refreshed output-timestep K/V within each hierarchy
Updated latent
Causal hierarchy pipeline
Rolling Forcing
Clean history and bidirectional live K/V
All active K/V if distributed
Joint mixed-noise denoising
Wave Forcing
Clean history and causal live K/V
Cleaner-to-noisier K/V and latent
Directed wavefront
Appendix
Table 9: Conditioning and exchanged state in different generation schedules. Live K/V is produced by active chunks at the current Transformer layer.
Method
Partitioned object
Main communication
Main constraint
Tensor parallelism
Hidden/head dimensions
Per-operator collectives
Small local GEMMs at high degree
Ulysses CP
Tokens to attention heads
Two All-to-All phases
Head divisibility and per-layer collectives
Ring Attention
K/V sequence blocks
Multi-round K/V circulation
Enough tile compute to hide P2P
Chunk-owned sync
Active query chunks
Layer K/V All-Gather
Global layer barrier
Wave Parallelism
Chunk–stage tasks
One-way K/V and latent
Full-model replicas and sufficient overlap slack
Appendix
Table 10: Resource-partitioning strategies, communication patterns, and constraints. TP and CP preserve the chosen graph, while WP schedules the fixed WF graph.
Parameters
Layers
Heads
Hidden dim.
FFN dim.
1.3B
30
12
1536
8960
14B
40
40
5120
13824
3B
32
18
2304
10752
5B
30
24
3072
14336
7B
29
30
3840
15360
9B
26
36
4608
18432
Appendix
Table 11: Model configurations for the parameter-scale comparison. The upper group contains the standard Wan2.1 architectures, and the lower group contains the synthetic architectures.
Streaming video diffusion models have made substantial progress toward interactive and dynamic world simulation, but the nested autoregressive and denoising loops of conventional next-frame generation hinder real-time deployment. Recent rolling-window methods pipeline denoising across multiple consecutive frames at different noise levels, improving throughput and long-horizon stability. However, they tokenize every state at the same fine spatial granularity, leaving substantial noise-dependent redundancy in the joint denoising window. We propose Ms.Forcing, an efficient streaming video generation paradigm that adapts spatial granularity to each state's noise level. Its Multi-Scale Patchification (MSP) assigns coarser patches to noisier states, reducing the active-window token count by 45%, while Multi-Scale Self-Attention (MSSA) matches the density of visible non-sink keys and values to each query scale to further reduce attention cost. Because both schedules are fixed by window position, Ms.Forcing retains a static, hardware-friendly computation graph. We further introduce Homogeneous-Noise-Level DMD (H-DMD), which assembles each fake video from clean predictions sharing the same source noise level, thereby reducing the mismatch between DMD training sequences and inference-time rollouts. The multi-scale design helps offset the additional training cost of backpropagating through overlapping windows. We include both quantitative and qualitative experiments to show that Ms.Forcing reaches 22.84 FPS on a single H200 GPU, 39.6% faster than Rolling Forcing, while significantly improving VBench scores in both short video and long video generation setting.
Zekun Li, Xiaoyan Cong, Hongyu Li +5
1Brown University · 2Massachusetts Institute of Technology · 3Meta
Few-step autoregressive video generators enable streaming generation, yet they run every token through the final denoising step, regardless of how much refinement it needs. We find this uniform computation is often unnecessary: many tokens already closely match their eventual full-step outputs in the initial steps, while others require refinement until the end. We introduce Speculative Forcing, a framework that treats intermediate clean estimates as drafts and predicts when individual tokens can be safely accepted early. Once accepted, a token is frozen as its final output and undergoes no further denoising, while its cached keys and values remain available as context for the remaining tokens. We explore two alternative verifier designs: a training-free one based on similarity to clean content and stability across steps, and a learned one that reads the frozen generator's intermediate features to estimate each token's remaining gap to its full-step output. Across short-video, long-video, and world generation, Speculative Forcing preserves the quality of full-step sampling while substantially reducing computation. On LongLive, it matches the four-step generator with ~1.7 denoising steps per token, running over 1.5x faster at 41.7 and 37.8 fps. On Lingbot-World 2.0, where camera motion continually changes the visible content, it maintains quality with ~2.5 denoising steps per token and runs 1.35x faster at 24.5 fps.
Diffusion Transformer (DiT)-based video generation models inherently suffer from bottlenecks in long video synthesis and real-time inference, which can be attributed to the use of full spatiotemporal attention. Specifically, this mechanism leads to explosive O(N^2) memory consumption and high first-frame latency. To address these issues, we implement system-level inference optimizations for a causal autoregressive video generation pipeline. We adapt the Self-Forcing causal autoregressive framework to sequence parallel inference and implement a sequence-parallel variant of the causal rotary position embedding which we refer to as Causal-RoPE SP. This adaptation enables localized computation and reduces cross-rank communication in sequence parallel execution. In addition, computation and communication pipelines are optimized through operator fusion and RoPE precomputation. Experiments conducted on an eight GPU A800 cluster show that the optimized system achieves comparable generation quality, sub-second first-frame latency, and near real-time inference speed. For generating five second 480P videos, a 1.58x speedup is achieved, thereby providing effective support for real-time interactive applications.