Scaling video diffusion transformers is fundamentally bottlenecked by two compounding costs: the expensive quadratic complexity of attention per step, and the iterative sampling steps. In this work, we propose EFlow, an efficient few-step training framework, that tackles these bottlenecks simultaneously. To reduce sampling steps, we build on a solution-flow objective that learns a function mapping a noised state at time t to time s. Making this formulation computationally feasible and high-quality at video scale, however, demands two complementary innovations. First, we propose Gated Local-Global Attention, a token-droppable hybrid block which is efficient, expressive, and remains highly stable under aggressive random token-dropping, substantially reducing per-step compute. Second, we develop an efficient few-step training recipe. We propose Path-Drop Guided training to replace the expensive guidance target with a computationally cheap, weak path. Furthermore, we augment this with a Mean-Velocity Additivity regularizer to ensure high fidelity at extremely low step counts. Together, our EFlow enables a practical from-scratch training pipeline, achieving up to 2.5x higher training throughput over standard solution-flow, and 45.3x lower inference latency than standard iterative models with competitive performance on Kinetics and large-scale text-to-video datasets.
Figures & tables
Figure 2: (a) Feed-forward latency vs. token number for different attention designs. Softmax attention (Wan 2.1) scales poorly as the sequence length grows while our GLGA ( EFlow ) is significantly faster than softmax and remains competitive with sparse/linear baselines. Applying 75% token dropping on GLGA yields the lowest latency and the most favorable scaling. (b) 480p inference latency for our model and recent video generators. Our model achieves 1.4 × and 45.3 × faster diffusion inference over baselines by combining an efficient backbone with few-step sampling.
Standard (Wan 2.1, Wan2.2)
Efficient (Sana, S 2 DiT)
Efficient (VSA, SLA)
Few-step (DMD, rCM)
Ours ( EFlow )
Attention architecture
Type
Softmax
Linear+Conv
Sparse softmax
Softmax
GLGA
Complexity
O(N2)
O(ND)
O(N′2)
O(N2)
O(N(D+W))
Token-Droppable
✓
✗
✗
✓
✓
Training pipeline
No Teacher Model
✓
✓
✗
✗
✓
Table 1: Comparison of our method with VideoDiT families. We summarize attention complexity, training pipeline requirements (reliance on teacher or auxiliary models like discriminator), and inference cost via number of function evaluations (NFEs). N , N′D , and W denote token count, sparse token number, dimension, and local window size, respectively. The “ ×2 ” indicates model evaluations for classifier-free guidance.
Figure 3: Overview of our VideoDiT architecture . (a) The network processes video tokens through stacked DiT blocks, utilizing long skip connections to stabilize information flow when tokens are aggressively dropped. (b) Given a DiT block with Gated Local–Global Attention (GLGA) module, input tokens are projected into shared queries, keys, and values. These features are processed by two parallel mechanisms: a linear attention branch for efficient global context and a sliding-window attention branch for expressive local detail. Finally, an input-aware gating ( g ) adaptively fuses the global ( Oglobal ) and local ( Olocal ) outputs on per-token basis.
Figure 4: End-to-end efficiency gains from our framework. (a) Training efficiency measured as throughput (iterations/sec). Relative to the standard flow-matching baseline, introducing the solution-flow (SoFlow) objective and the MVA regularizer increases per-iteration overhead and therefore reduces throughput. In contrast, our efficient backbone, PDG training, and token dropping provide consistent speedups, improving throughput by up to 1.6× and 2.5× over the baselines. (b) Inference efficiency. Our overall framework substantially reduces end-to-end inference time relative to the softmax+flow-matching baseline ( 147→10.09 s).
Figure 5: Systemic comparison on Kinetics 700. (a) Training loss vs. iterations comparing the softmax-attention baseline, Sana-Video attention, and our GLGA module. (b) FVD vs. training compute across various model scales for softmax, Sana-Video attention, sparse-linear attention (SLA), and GLGA. GLGA consistently achieves lower loss and lower FVD at matched compute, and its advantage persists as model scale increases; SLA become unstable and diverge under the same training setup.
Figure 6: Qualitative comparison on Kinetics-700 on 400K training iterations. Our complete framework (Ours+PDG-SFM) successfully generates highly detailed, temporally consistent videos in only 4 inference steps. In contrast, baselines trained with standard Flow Matching (FM)—including standard softmax, and Sana-video’s attention—struggle to resolve vivid details and exhibit severe blurring or structural distortion despite using 50 steps.
Attn.
Objective
MVA
Token drop
Params
NFE ↓
FVD ↓
Base1
Softmax
FM
✗
✗
457M
40
188.9
Base2
Sana
FM
✗
✗
607M
40
217.7
Base3
Sana
FM
✗
✓
607M
40
456.1
Ours1
GLGA
FM
✗
✗
529M
40
191.3
Ours2
GLGA
FM
✗
✓
529M
40
189.7
Ours3
GLGA
PDG-SFM
✗
✓
529M
4
185.2
Table 2: FVD comparison against baselines. All models are trained for 400K iterations.
Table 8
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
EFlow -B (Fig. 5 , Tab. 3 , 4 )
EFlow -XL (Fig. 5 – 9 , Tab. 2 )
EFlow -XXL (Fig. 5 )
EFlow -1.5B (Fig. 4 , Tab. 6 )
Params.
128M
529M
3B
1.5B
Patch size
2×2×1
2×2×1
2×2×1
2×2×1
Total Num. blocks
14
28
36
28
Ne
2
3
4
3
Nm
10
22
28
22
Nd
2
3
4
3
Appendix
Table 5: Hyperparameters used in our experiments. Kinetics-700 models (B/XL/XXL) and Text-to-Video model (1.5B).
Figure 7: Training loss over 400K iterations. Our EFlow (red) converges rapidly and remains stable throughout training. In contrast, the Video Sparse Attention (VSA) baseline (blue) exhibits pronounced optimization instability, with frequent loss spikes early in training and reduced convergence reliability under the same setup. These results suggest that VSA is more sensitive to training configurations, whereas EFlow trains robustly without specialized stabilization or hyperparameter tuning.
Figure 8: Visual ablation of token-drop training. We compare models trained (a) entirely on dense grids without token dropping, and (b) utilizing our aggressive 75% token-dropping framework. Despite the massive reduction in training compute, the model trained with token dropping rigorously preserves spatial fidelity, fine-grained details, and overall temporal consistency, demonstrating that our sparse training recipe does not degrade generative quality.
Figure 9: Visual ablation of Mean-Velocity Additivity (MVA). Both examples use 4-step sampling. (a) Without MVA, integration errors accumulate across steps, leading to distortion, motion blur, and structural collapse. (b) With MVA, enforcing global semigroup consistency improves sharp structure, consistent subject identity and motion.
Methods
NFE
Latency (s)
Speedup
#Params (B)
Evaluation scores ↑
Total
Quality
Semantic
Multi-step DiT
Wan2.1
40 × 2
732
1.0 ×
14B
82.34
83.77
76.65
Wan2.1
40 × 2
147
4.97 ×
1.3B
82.11
83.77
75.46
SANA-Video
40 × 2
79.03
9.26 ×
2B
81.93
84.11
73.23
Few-step DiT with teacher-distillation
Appendix
Table 6: VBench results for 480p text-to-video generation. All baselines are re-evaluated using the Diffusers library with our augmented prompts for consistent comparison. ⋆ denotes models specifically fine-tuned on Wan-14B-generated images. † denotes unofficial checkpoints due to restricted access to the official open-source release.
Figure 10: Qualitative comparison on Kinetics-700 on 400K training iterations.
Figure 11: Qualitative comparison on Kinetics-700 on 400K training iterations.
Figure 12: Qualitative comparison on Kinetics-700 on 400K training iterations.
Figure 13: Qualitative comparison on 480p text-to-video . Left column prompt: In the video, a man is seen in a kitchen, preparing food. He is wearing a white shirt and is focused on his task. The man is holding a large metal bowl filled with what appears to be a mixture of ingredients, possibly a salad or a mixture for a dish. He is using a spoon to mix the contents of the bowl, ensuring that everything is well combined. Right column prompt: In the video, a man is standing on a sandy beach, wearing a black hoodie and sunglasses. He is speaking to the camera, gesturing towards a white Chevrolet truck parked in front of a house. The truck is parked on a concrete driveway, and the house has a white exterior with a wooden deck and stairs leading up to the front door. The sky is clear and blue, indicating a sunny day. The man appears to be enjoying his time at the beach, and the truck is parked in a way that suggests it is ready for a road trip.
Figure 14: Qualitative comparison on 480p text-to-video . Left column prompt: "The video captures a thrilling mountain biking experience in a dense forest. The perspective is from the rider’s point of view, providing an immersive experience of the journey. The rider is seen maneuvering through a narrow, winding trail covered with fallen leaves and twigs, indicating a fall season setting. The trail is surrounded by tall trees with green foliage, creating a canopy overhead. The rider’s hands are firmly gripping the handlebars, guiding the bike through the uneven terrain." Right column prompt: "The video captures a serene and picturesque scene of a small waterfall nestled within a lush, green forest. The waterfall cascades down a rocky ledge, surrounded by a variety of vibrant green plants and foliage. The water flows smoothly, creating a gentle, soothing sound that adds to the tranquil atmosphere. The surrounding vegetation includes large, broad leaves and smaller, delicate plants, creating a dense and verdant environment."
Figure 15: Qualitative comparison on 480p text-to-video . Prompt: CG game concept digital art, a person sitting at a wooden table, focused intently on a sketchbook. They are wearing comfortable casual clothes, such as a plain t-shirt and jeans. The person has short, messy brown hair and a gentle smile, holding a pencil in their hand. The sketchbook contains detailed drawings of various creatures, including dragons, mythical beasts, and futuristic machines. Soft lighting casts shadows on the walls behind them, creating a cozy atmosphere. The person is drawing with a calm and deliberate pace, occasionally pausing to consult reference materials.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
Few-step video generation has been significantly advanced by consistency distillation. However, the performance of consistency-distilled models often degrades as more sampling steps are allocated at test time, limiting their effectiveness for any-step video diffusion. This limitation arises because consistency distillation replaces the original probability-flow ODE trajectory with a consistency-sampling trajectory, weakening the desirable test-time scaling behavior of ODE sampling. To address this limitation, we introduce AnyFlow, the first any-step video diffusion distillation framework based on flow maps. Instead of distilling a model for only a few fixed sampling steps, AnyFlow optimizes the full ODE sampling trajectory. To this end, we shift the distillation target from endpoint consistency mapping (zt→z0) to flow-map transition learning (zt→zr) over arbitrary time intervals. We further propose Flow Map Backward Simulation, which decomposes a full Euler rollout into shortcut flow-map transitions, enabling efficient on-policy distillation that reduces test-time errors (i.e., discretization error in few-step sampling and exposure bias in causal generation). Extensive experiments across both bidirectional and causal architectures, at scales ranging from 1.3B to 14B parameters, demonstrate that AnyFlow achieves performance matches or surpasses consistency-based counterparts in the few-step regime, while scaling with sampling step budgets.
Yuchao Gu, Guian Fang, Yuxin Jiang +4
1NVIDIA · 2Show Lab, National University of Singapore · 3MIT
Video generation models based on Diffusion Transformers (DiTs) have achieved remarkable performance in video synthesis, yet they suffer from high inference latency and computational costs due to the quadratic complexity of 3D attention. Existing acceleration methods primarily reduce computational complexity within each individual denoising steps through techniques such as sparse attention and KV-caching. However, they rigidly adhere to the inherent constraint of the standard diffusion pipeline: every frame in the target video sequence must be subjected to a complete, dense denoising process across all diffusion timesteps. We observe that due to the corresponding contents and motions among adjacent frames, when keyframes with critical semantic transitions are anchored, the intermediate states of others often follow more predictable trajectories, which indicates that such uniform, dense denoising process is inherently redundant for natural video data. To this end, we introduce RhymeFlow, a training-free framework that decouples the denoising trajectories of different frames. Specifically, we first identify a sparse set of pivotal key frames that dominate the latent semantic evolution. Then, only these keyframes undergo dense, step-by-step denoising to ensure structural integrity, while non-keyframes progressively skip denoising steps to minimize computational cost. Since skipped intermediate states of non-keyframes break the temporal coherence in keyframe denoising steps, leading to visual degradation, we further introduce a latent trajectory projection module, which enables keyframes to interact with a complete and temporally consistent sequence representation. Extensive experiments on current DiT-based video generation models demonstrate our method outperforms existing baselines with higher inference speed and better visual quality.
Generation in video diffusion or flow models is computationally expensive due to the slow and iterative sampling process. Current state-of-the-art (SOTA) acceleration methods heavily rely on variational score distillation (VSD) and adversarial losses to distill diffusion models into few-step generators. Albeit achieving high-quality video generation, these training losses are notoriously hard to optimize and suffer from mode collapse, leading to loss of video diversity and lack of motion. In this paper, we introduce Parallel Decoding Distillation (PDD), a simplified and scalable trajectory-based distillation method for fast inference of diffusion and flow matching models. Our architecture and training procedure are compatible with any pre-trained model and support sampling with a varying number of function evaluations (NFE). PDD accelerates generation by predicting multiple denoising steps per network evaluation. Conceptually, it learns a representation of the mean velocity without regressing its derivative using JVPs or finite-difference approximations. Our method achieves SOTA performance with 4-8 NFE on LTX-2.3 Text-to-Video/Audio, Wan 14B Text-to-Video, and Qwen-Image Text-to-Image. Moreover, PDD presents a significant improvement in generated video diversity.