Bidirectional video diffusion models denoise entire videos in parallel, yet when trained on effectively unlimited in-distribution data from procedural generators, continue to violate physical laws and simple symbolic rules. We introduce Serial-to-Parallel Diffusion (S2PD), which performs autoregressive diffusion at high noise before switching to parallel diffusion at low noise. The autoregressive phase provides the serial computation needed to coordinate interdependent events and produce valid state transitions while the parallel phase jointly refines the entire video and reduces sampling time relative to fully serial generation. We implement S2PD with two architectures: a pixel-space diffusion transformer trained from scratch and a pretrained video model adapted through LoRA fine-tuning with causal attention. Across games, physical simulations, and real video, S2PD follows rules more reliably than matched bidirectional baselines and generates videos with greater temporal stability and sampling efficiency than other serial methods.
Figures & tables
Figure 1: Serial outperforms parallel generation. Our method better preserves game rules, physical dynamics, and object consistency across all datasets. Red overlays localize errors in the generated samples.
Figure 2: Attention patterns across methods and our sampling strategy. (a–c) Attention masks and noise levels for Bidirectional, cDF, and BC. (d) Serial denoising followed by parallel refinement. (e) Token order within a temporal slice. (f) Multiple block sizes used by Ours during training.
Table 2: Main comparisons. Serial methods outperform the bidirectional baseline and perform similarly to each other across all datasets. cDF and BC use block sizes of 64. Ours uses a block size of 8 for discrete datasets and 64 for continuous and video datasets. Best is marked in blue and worst is marked in red.
Figure 6: Qualitative comparisons with the Wan-5B-LoRA architecture. Serial methods perform similarly and produce more coherent video than the bidirectional baseline on MPMWorlds and Rubik’s Cube Real datasets.
Figure 7: Spatiotemporal slices expose temporal flickering. Ours and ground truth show clean vertical lines while cDF and BC show subtle patchy artifacts.
Table 6
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
DiT-B
Wan-5B-LoRA
Architecture
Depth, width, heads
12, 768, 12
30, 3072, 24
Tubelet size
4×32×32 pixels
1×2×2 latents
Objective
x-pred
v-pred
Trainable parameters
132.7M
80.6M / 5.00B frozen
Bottleneck dimension
128
—
Appendix
Table 7: Detailed hyperparameters for S2PD with DiT-B and Wan-LoRA-5B. Shared entries span both columns. Architecture-specific parameters are listed in separate columns.
BC
cDF
Ours
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Kubric MOVi-A
176.1
141.2
153.8
182.6
278.7
366.2
168.5
165.8
175.5
240.0
162.1
144.6
139.2
137.0
137.7
Kubric MOVi-C
265.5
240.3
286.8
469.9
913.0
1353.9
376.5
345.4
454.1
894.9
319.9
250.4
227.0
231.7
271.3
MPMWorlds
48.2
43.9
49.0
63.3
115.5
101.1
43.2
47.3
58.8
95.5
50.6
41.8
42.1
42.5
43.1
Rubik’s Cube Real
69.4
70.9
68.2
68.3
66.9
96.2
70.8
71.9
71.8
74.9
68.6
66.6
65.6
64.4
65.0
Appendix
Table 8: CD-FVD ↓ with Wan-5B-LoRA at block size 64. BC and cDF vary the re-noising used to resist autoregressive drift and Ours varies the transition noise level.
BC
cDF
Ours
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Kubric MOVi-A
241.0
186.1
210.4
202.2
234.0
582.1
249.3
242.7
266.2
254.7
221.3
176.0
181.0
162.2
171.1
Kubric MOVi-C
264.7
242.6
212.6
236.8
311.1
1115.3
302.7
231.1
255.9
333.2
289.3
228.1
199.0
186.2
214.1
MPMWorlds
35.7
32.0
33.3
40.0
56.1
141.8
33.2
32.6
37.8
49.9
43.6
32.9
31.1
33.2
33.3
Rubik’s Cube Real
105.8
100.3
95.2
91.6
89.6
121.9
95.5
94.2
90.8
90.4
110.6
102.3
102.8
100.4
99.6
Appendix
Table 9: FVD ↓ with Wan-5B-LoRA at block size 64. BC and cDF vary the re-noising used to resist autoregressive drift and Ours varies the transition noise level.
Figure 8: Discrete datasets. Qualitative comparison on Conway, Chess, 2048, Fifteen Puzzle, Tetris, Snake, and Rubik’s Cube 3D. For each dataset, the top row shows Bidirectional and the bottom row shows Ours. We show select frames from the ten-second video samples.
Figure 9: Continuous datasets. Qualitative comparison on Double Pendulum, Three Body, Colliding Balls 2D, Colliding Balls 3D, and Pong. For each dataset, the top row shows Bidirectional and the bottom row shows Ours. We show select frames from the ten-second video samples.
Figure 10: Video datasets. Qualitative comparison on Kubric MOVi-A, Kubric MOVi-C, MPMWorlds, and Rubik’s Cube Real. For each dataset, the top row shows Bidirectional and the bottom row shows Ours. We show select frames from the ten-second video samples.
Recent progress in large-scale generative models has substantially advanced video generation, yet existing methods remain constrained by a rigid inference paradigm. Bidirectional diffusion models excel at global coherence and visual fidelity but suffer from slow inference, while autoregressive models offer efficient and streaming generation at the cost of long-range consistency and exposure bias. We introduce Flex-Forcing, a unified training and inference framework that enables a video diffusion model to seamlessly operate under both bidirectional and autoregressive generation regimes. The core idea is a flexible chunking mechanism jointly defined over the temporal axis and denoising steps. This design allows the model to (1) perform flexible chunking according to different device budgets, (2) perform bidirectional inference across chunks for global structure planning, while generating frames autoregressively within each chunk for efficient and fine-grained synthesis, and (3) perform any-order, any-timestep autoregressive generation without the strict causal constraint. Extensive experiments on multiple video generation benchmarks demonstrate that Flex-Forcing achieves consistently better video quality, long-video stability than strong baselines with a rigid inference schedule, while offering faster inference.
Xinyin Ma, Julius Berner, Chao Liu +3
Electrical and Computer Engineering, National University of Singapore, Singapore · NVIDIA Research, California, USA
When one ball strikes another, then another, video models should predict the consequences of each bounce. In controlled experiments on multi-ball hard-sphere dynamics, we find that the performance of standard bidirectional video diffusion degrades as the causal chain lengthens, even when provided more denoising steps. In a length-matched single-ball control, where ball-ball interactions are absent, the degradation largely disappears, isolating dependent-event structure rather than video length as the cause. Across intervention studies, methods that increase effective serial computation improve performance disproportionately, including autoregressive/blockwise generation and architectural depth. We identify this pattern as the seriality gap: a mismatch between tasks requiring growing serial computation and video diffusion models whose denoising loop does not provide scalable serial compute. We then prove that, for deterministic video prediction, denoising steps do not add serial computation beyond the backbone, indicating a structural obstacle for video diffusion on serial reasoning and simulation tasks.
Generation in video diffusion or flow models is computationally expensive due to the slow and iterative sampling process. Current state-of-the-art (SOTA) acceleration methods heavily rely on variational score distillation (VSD) and adversarial losses to distill diffusion models into few-step generators. Albeit achieving high-quality video generation, these training losses are notoriously hard to optimize and suffer from mode collapse, leading to loss of video diversity and lack of motion. In this paper, we introduce Parallel Decoding Distillation (PDD), a simplified and scalable trajectory-based distillation method for fast inference of diffusion and flow matching models. Our architecture and training procedure are compatible with any pre-trained model and support sampling with a varying number of function evaluations (NFE). PDD accelerates generation by predicting multiple denoising steps per network evaluation. Conceptually, it learns a representation of the mean velocity without regressing its derivative using JVPs or finite-difference approximations. Our method achieves SOTA performance with 4-8 NFE on LTX-2.3 Text-to-Video/Audio, Wan 14B Text-to-Video, and Qwen-Image Text-to-Image. Moreover, PDD presents a significant improvement in generated video diversity.