Few-step distillation offers efficient generation with diffusion and flow models, but preserving quality with only a few sampling steps remains challenging. Distribution-matching methods via score distillation provide a strong foundation, yet they rely on accurately tracking the student's changing distribution. At intermediate generation steps in few-step sampling, student-score estimation can be difficult, and commonly used estimators may describe a different distribution from the one used to update the student. Flow matching allows a more flexible approach to supervision through velocity fields constructed from the student's sampling process. Building on this perspective, we introduce Velocity Adversarial Distillation (VAD), which uses a learned critic to compare student and teacher fields and turn their disagreement into feedback for learning. VAD unifies score distillation, Fisher-based matching, and moment matching, also explaining existing heuristic MMD updates and delivering new training objectives. We further develop VA-TDM, which follows trajectory distribution matching: providing supervision for partial denoising along the student trajectory. Beyond this, VA-TDM combines transition-aware online velocity learning and dual-form VAD supervision across intermediate student transitions. Extensive results show that VA-TDM is powerful for learning few-step generators, achieving state-of-the-art four-step performance in both text-to-image and text-to-video generation across various backbones and scaling effectively to 33B MiniMax-H3. In particular, our VA-TDM-H3 notably outperforms the few-step Fast-H3 and is closer to the original many-step H3 according to human preference.
Figures & tables
Figure 1: Human preference on video synthesis.
Figure 2: Comparison between detached critic and non-detached critic. We found that the non-detached critic leads to unstable training and collapse.
Model
NFE
Verifiable
ImgQ
Preference
Div.
GenEval
OCR
Aest.
ImgRwd
PickScore
HPSv2
HPSv3
LPIPS
SD3.5-Medium (teacher, CFG=4.5)
50 × 2
0.65
0.58
5.37
0.86
0.8628
0.2805
9.78
0.61
SD3.5-Medium (teacher, CFG=7)
50 × 2
0.67
0.62
5.37
0.96
0.8640
0.2854
9.87
0.60
DMD2
4
0.60
0.53
5.55
1.00
0.8648
0.2977
11.18
0.53
D-DMD
4
0.55
0.56
5.51
0.99
0.8567
0.2897
10.93
0.57
DPDMD
4
0.62
0.33
5.28
0.80
0.8530
0.2707
8.13
0.59
Table 1: Systematic evaluation. Best / second-best few-step results per backbone are bold / underlined .
Figure 3: Qualitative comparison among VA-TDM and competing baselines.
Configuration
Verifiable
ImgQ
Preference
Div.
GenEval
OCR
Aest.
ImgRwd
PickScore
HPSv2
HPSv3
LPIPS
λm=1 (A-form only)
0.64
0.55
5.53
1.00
0.8671
0.2949
11.20
0.58
λm=0 (B-form only)
0.61
0.57
5.57
0.99
0.8665
0.2987
11.47
0.59
λm=0 w/o transition-aware field
0.58
0.53
5.56
0.99
0.8655
0.2964
11.13
0.56
λm=0.8 w/o shared backbone
0.62
0.56
5.55
0.99
0.8669
0.2971
11.33
0.57
λm=0.8
0.63
0.58
5.56
1.01
0.8674
0.2983
11.47
0.58
Table 2: Ablation study. Best / second-best results are bold / underlined .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).
Figure 5: Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).
Figure 6: Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).
Figure 7: Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).
Figure 8: Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).
Few-step distillation reduces the sampling cost of diffusion and flow models, but existing approaches often require auxiliary score models or specialized architectures. We show that simple on-policy velocity regression already provides a strong foundation for 4-NFE generation. To represent teacher velocity variation within each coarse transition, we introduce Pivot, a training-only residual proxy that separates local corrections from the student's interval-mean prediction. We further repel the student from weaker teacher predictions, providing a lightweight mechanism for avoiding undesirable modes. The resulting framework, Simplified Matching Distillation (SMD), remains entirely regression-based, requires neither online score estimation nor adversarial training, and leaves the deployed generator unchanged. We evaluate SMD on image generation and editing, 3D generation, video synthesis, and joint audio--video generation, scaling to 33 billion parameters. At 4-NFE, SMD outperforms strong few-step baselines and surpasses multi-step teachers on several benchmarks. It also supports cross-model, multi-teacher, and causal-video distillation.
Recent advances in few-step diffusion distillation have enabled efficient image generation, yet aligning these models with human preferences remains challenging. We propose Reward-Tilted Distribution Matching Distillation (RTDMD), a two-stage framework that unifies distribution matching distillation with reward-guided reinforcement learning for few-step flow generators. We show that minimizing the KL divergence to a reward-tilted teacher distribution naturally decomposes into a distribution matching term and a reward maximization term. In the first stage, we introduce Ambient-Consistent Distribution Matching Distillation (AC-DMD), which performs subinterval-wise distribution matching and augments the fake score objective with a consistency regularizer to help the fake score model track the shifting generator distribution under limited updates. In the second stage, we jointly optimize both terms: for the reward maximization term, we derive a hybrid policy gradient that combines a GRPO-style estimator for the stochastic intermediate transitions with direct reward backpropagation through the deterministic final step, and further introduce step-subset GRPO (SubGRPO) to reduce variance. Experiments on SD3, SD3.5, and FLUX.2 demonstrate that RTDMD establishes new state-of-the-art results across preference, aesthetic, and compositional metrics with only 4 inference steps, outperforming previous few-step text-to-image generation methods. Code and models are available at https://github.com/Harahan/RTDMD.
Yushi Huang, Xiangxin Zhou, Ruoyu Wang +3
Tencent Hunyuan · Hong Kong University of Science and Technology · Westlake University
Video creation spans text-to-video (T2V), image-to-video (I2V), and condition-based generation, yet video diffusion models remain costly because they repeatedly evaluate large backbones during sampling. Distribution matching distillation (DMD) reduces this cost, but its reverse Kullback--Leibler (KL) objective can provide unstable or incomplete guidance when the student and teacher distributions have limited overlap. VDOT addressed this issue by adding optimal transport distillation (OTD), whose explicit coupling supplies geometric directions for condition-based generation. Balanced OTD, however, performs full-mass matching between the spatial tokens of each corresponding student--teacher frame pair. This assumption weakens for T2V and I2V, where one condition admits many valid outputs and spatial content need not align across different realizations. We present VDOT++, a unified distillation framework that applies the same training recipe separately to generators for the three task families. It makes OTD robust to output diversity through an asymmetric unbalanced formulation that allows unreliable student tokens to carry less mass while maintaining coverage of the teacher tokens. An ℓ1 ground cost further replaces mean-based aggregation with a more mode-preserving weighted median that limits the influence of distant transport targets. The two changes respectively determine whom to match and how the selected targets should be aggregated. We additionally combine distribution matching and adversarial refinement through sequential backward passes, and exploit the decoupled score networks for cross-scale distillation, where larger score networks improve a compact generator. Experiments on UVCBench, VBench, VBench-I2V, and the VACE benchmark show that the resulting four-step generators are competitive with many-step teachers and strong few-step baselines across all three task families.
Yutong Wang, Xingtong Ge, Enhuai Liu +6
University of Sydney · Hong Kong University of Science and Technology · Shanghai AI Laboratory +1