Distribution Matching Distillation~(DMD) enables strong few-step diffusion generation by matching the student distribution to a pretrained teacher. Existing improvements to DMD often augment its supervision or modify its optimization machinery. We ask a complementary question: can DMD diagnose its own updates using signals already produced during training? DMD inherently computes the discrepancy between a pretrained teacher and a fake model tracking the student distribution, but conventionally uses it primarily to determine the distribution matching direction. We find that the magnitude of this teacher--fake disagreement provides additional diagnostic information. Across extensive experiments, large disagreement in the high-noise regime consistently coincides with substantially higher-energy DMD updates. This reveals a dual role of disagreement: it determines not only where the student should move, but also provides an intrinsic signal of how aggressively the update should be applied. Based on this observation, we introduce Disagreement-Aware Distribution Matching Distillation (DA-DMD), which activates only in the high-noise regime and continuously modulates the update strength according to teacher--fake disagreement. DA-DMD reuses quantities already available during DMD training and requires no additional model, evaluator, or inference-time modification. We show that it exactly recovers vanilla DMD outside the selected region, reduces update energy only through selected samples, and exhibits redescending influence under extreme disagreement. Across multiple diffusion backbones, DA-DMD consistently improves few-step generation at the same inference cost. On SD3.5-L, DA-DMD improves HPSv2 from 28.23 to 30.93, while achieving a 64.4% average win rate over DMD2.
Figures & tables
Figure 1: Teacher–fake disagreement diagnoses DMD update aggressiveness. Within the predefined high-noise region H , larger normalized disagreement is consistently associated with higher-energy raw DMD updates across DIGITS, CIFAR-10, and CIFAR-100. DA-DMD continuously attenuates these updates according to disagreement. Shaded regions highlight the high-disagreement tail for visualization only and do not correspond to an algorithmic threshold.
Figure 2: High-energy DMD updates concentrate at high noise. During SD3.5-M distillation, raw DMD update energy rises sharply in the high-noise region H , where DA-DMD activates disagreement-aware attenuation.
Figure 3: Overview of DA-DMD. (a) In the active high-noise region, DA-DMD replaces the raw teacher-directed correction with a disagreement-aware proximal interpolation between the fake and teacher predictions, yielding an attenuated DMD update with multiplier ωα(r) . (b) Large teacher–fake disagreement can induce disproportionately large raw DMD corrections. (c) DA-DMD shortens these corrections toward an adaptive proximal target, reducing the influence of high-disagreement updates while preserving lower-disagreement updates.
Backbone
Method
PickScore ↑
ImageReward ↑
HPSv2 ↑
AES ↑
VQ-R1 ↑
Avg. WR ↑
SANA
DMD2 ( Yin et al., 2024a )
22.53
95.05
29.67
5.820
4.580
-
DA-DMD (Ours)
22.56 +0.03
101.58 +6.53
29.87 +0.20
5.945 +0.125
4.643 +0.063
56.0%
SANA-1.5
DMD2 ( Yin et al., 2024a )
22.53
98.57
29.06
5.858
4.525
-
DA-DMD (Ours)
22.58 +0.05
102.09 +3.52
29.35 +0.29
5.916 +0.058
4.549 +0.024
57.0%
SD3-M
DMD2 ( Yin et al., 2024a )
22.33
98.56
30.00
5.463
4.534
-
Hyper-SD3 ( Ren et al., 2024 )
21.99
67.67
26.30
5.305
4.292
28.8%
Table 1: Quantitative comparison on DrawBench. DA-DMD provides broad improvements over DMD2 across multiple diffusion backbones and evaluation metrics.
Figure 4: Sensitivity to the weighting coefficient α on SD3.5-M. Moderate weighting, particularly α=0.3 , provides the best overall trade-off across metrics on DrawBench and PartiPrompts.
Figure 5: Qualitative comparison with representative distillation baselines on SD3.5-M.
Figure 6: Effect of the intervention region on SD3.5-M. High-noise intervention provides the strongest overall trade-off across DrawBench and PartiPrompts, while global, middle-noise, and low-noise attenuation are generally less effective.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Teacher NFE
Teacher CFG
α
SANA
20
4.5
0.2
SANA-1.5
20
4.5
0.2
SD3-M
30
7.0
0.3
SD3.5-M
30
3.5
0.3
SD3.5-L
30
3.5
0.7
Appendix
Table 2: Main training hyperparameters. DA-DMD uses the same optimization and sampling configuration as DMD2, with model-specific teacher guidance and disagreement weighting coefficient α .
Figure 7: Selective disagreement-aware weighting in DA-DMD. (a) Outside the predefined high-noise region H , DA-DMD exactly preserves the vanilla DMD update. Within H , the retained update strength decreases continuously with normalized teacher–fake disagreement r , with α controlling its maximum value. (b) The resulting weighting leaves low-/mid-noise updates unchanged while substantially attenuating the high-energy updates concentrated in the high-noise regime.
Figure 8: Controlled analysis of disagreement-aware DMD optimization on CIFAR-10. Starting from the same initialization, raw DMD can be dominated by high-disagreement samples with large update magnitudes, leading to an imbalanced optimization trajectory. DA-DMD selectively attenuates these updates while retaining lower-disagreement corrections, resulting in improved target-mode matching. The bottom panels visualize the evolution of target-mode occupancy and the concentration of large raw DMD updates among high-disagreement samples.
Figure 9: Controlled analysis of disagreement-aware DMD optimization on DIGITS. Starting from the same initialization, raw DMD can be dominated by high-disagreement samples with large update magnitudes, leading to an imbalanced optimization trajectory. DA-DMD selectively attenuates these updates while retaining lower-disagreement corrections, resulting in improved target-mode matching. The bottom panels visualize the evolution of target-mode occupancy and the concentration of large raw DMD updates among high-disagreement samples.
Figure 10: Controlled analysis of disagreement-aware DMD optimization on CIFAR-100. Starting from the same initialization, raw DMD can be dominated by high-disagreement samples with large update magnitudes, leading to an imbalanced optimization trajectory. DA-DMD selectively attenuates these updates while retaining lower-disagreement corrections, resulting in improved target-mode matching. The bottom panels visualize the evolution of target-mode occupancy and the concentration of large raw DMD updates among high-disagreement samples.
Backbone
Method
PickScore ↑
ImageReward ↑
HPSv2 ↑
AES ↑
VQ-R1 ↑
Avg. WR ↑
SANA
DMD2
22.74
117.28
30.57
5.940
4.617
-
DA-DMD (Ours)
22.77 +0.03
119.58 +2.30
30.57 +0.00
5.956 +0.016
4.638 +0.020
53.6%
SANA-1.5
DMD2
22.85
122.79
30.31
6.022
4.553
-
DA-DMD (Ours)
22.90 +0.05
122.29 -0.50
30.44 +0.13
6.050 +0.029
4.575 +0.023
54.6%
SD3-M
DMD2
22.39
116.74
30.44
5.697
4.571
-
Hyper-SD3
22.24
92.07
27.87
5.530
4.418
30.0%
Appendix
Table 3: Quantitative comparison on PartiPrompts. DA-DMD yields broad improvements over the corresponding DMD2 baselines across different model families.
Backbone
Overall ↑
WR (%) ↑
DMD2
DA-DMD
Δ
SANA
14.68
15.89
+1.21
55.0
SD3.5-M
20.17
21.74
+1.57
55.4
SD3.5-L
15.93
22.74
+6.81
61.9
Appendix
Table 4: GenEval2 results across backbones. Δ denotes the absolute improvement over DMD2. WR is the pairwise win rate of DA-DMD against DMD2.
Backbone
Method
3 ↑
4 ↑
5 ↑
6 ↑
7 ↑
8 ↑
9 ↑
10 ↑
SANA
DMD2
42.03
32.25
14.39
11.27
6.48
5.53
3.22
2.23
DA-DMD
44.68 +2.65
32.11 -0.14
13.89 -0.50
13.08 +1.81
10.41 +3.93
7.08 +1.55
4.22 +1.00
3.37 +1.14
SANA-1.5
DMD2
53.69
43.07
20.28
18.02
6.95
9.27
5.63
3.43
DA-DMD
54.66 +0.97
44.20 +1.13
22.62 +2.34
17.97 -0.05
6.59 -0.36
8.84 -0.43
6.73 +1.10
3.98 +0.55
SD3.5-M
DMD2
50.01
39.16
26.65
18.27
8.44
7.75
7.90
3.14
DA-DMD
53.47 +3.46
36.63 -2.53
27.68 +1.03
19.05 +0.78
15.15 +6.71
9.02 +1.27
8.58 +0.68
4.32 +1.18
Appendix
Table 5: GenEval2 results across prompt complexity. DA-DMD provides broad gains across compositional complexity levels, with particularly strong improvements on challenging high-atom prompts, e.g., +13.04 and +9.56 on 7- and 8-atom prompts with SD3.5-L. Colored values denote absolute changes over DMD2.
Backbone
Method
Overall ↑
Overall S ↑
Overall L ↑
Basic S ↑
Basic L ↑
Attr. S ↑
Attr. L ↑
Rela. S ↑
Rela. L ↑
Reason S ↑
Reason L ↑
SANA
DMD2
60.72
60.1
61.3
75.8
77.3
79.3
79.6
84.1
84.6
63.4
67.6
DA-DMD
61.12 +0.40
60.4 +0.3
61.8 +0.5
75.7 -0.1
77.1 -0.2
79.1 -0.2
79.8 +0.2
84.2 +0.1
84.3 -0.3
63.4 +0.0
66.9 -0.7
SANA-1.5
DMD2
62.47
63.2
61.8
77.1
76.4
81.2
79.2
84.5
83.9
65.1
65.7
DA-DMD
62.84 +0.37
63.5 +0.3
62.2 +0.4
77.5 +0.4
76.6 +0.2
80.8 -0.4
79.1 -0.1
84.8 +0.3
84.7 +0.8
66.3 +1.2
66.1 +0.4
SD3-M
DMD2
63.33
63.8
62.8
77.8
77.8
83.5
82.5
82.3
83.1
66.4
66.9
DA-DMD
64.23 +0.90
64.5 +0.7
63.9 +1.1
78.0 +0.2
78.4 +0.6
83.9 +0.4
85.1 +2.6
82.0 -0.3
81.9 -1.2
66.9 +0.5
66.7 -0.2
Appendix
Table 6: TIIF-Bench overall and basic-category results. DA-DMD improves the overall instruction-following score across all five evaluated backbones, with the largest gain of +2.44 on SD3.5-L, while showing broadly positive improvements across short- and long-form categories.
Backbone
Method
Adv. S ↑
Adv. L ↑
Attr.+Rela. S ↑
Attr.+Rela. L ↑
Attr.+Rea. S ↑
Attr.+Rea. L ↑
Rela.+Rea. S ↑
Rela.+Rea. L ↑
SANA
DMD2
51.8
52.8
78.0
76.3
63.3
66.9
66.0
68.7
DA-DMD
52.3 +0.5
53.8 +1.0
78.2 +0.2
78.2 +1.9
63.5 +0.2
66.9 +0.0
67.0 +1.0
71.0 +2.3
SANA-1.5
DMD2
56.1
54.0
80.4
76.7
68.7
65.9
68.8
69.4
DA-DMD
56.3 +0.2
54.7 +0.7
80.6 +0.2
76.8 +0.1
69.8 +1.1
67.2 +1.3
68.4 -0.4
68.6 -0.8
SD3-M
DMD2
57.1
55.4
78.0
78.7
63.9
65.7
63.2
66.9
DA-DMD
58.2 +1.1
57.0 +1.6
78.8 +0.8
79.2 +0.5
65.5 +1.6
66.9 +1.2
65.2 +2.0
68.6 +1.7
Appendix
Table 7: TIIF-Bench advanced compositional results. DA-DMD generally improves complex compositional instruction following across model backbones, with particularly strong gains on SD3.5-L, including +2.4 and +3.3 on advanced short- and long-form prompts. Colored values denote absolute changes over DMD2.
Figure 11: Qualitative comparison with representative distillation baselines on SD3.5-M. DA-DMD demonstrates stronger prompt adherence and visual consistency across challenging cases involving counting, text rendering, fine-grained attributes, and object identity, compared with DMD, SiD-Adv, SiD-DiT, SenseFlow, and DP-DMD.
Figure 12: Qualitative comparison on SD3-M. DA-DMD improves prompt adherence over DMD2 across diverse compositional challenges, including object counting, spatial placement, relative relations, color control, and structured multi-object generation.
Figure 13: Qualitative comparison on SD3.5-M. DA-DMD improves prompt adherence over DMD2 across diverse compositional challenges, including object counting, spatial placement, relative relations, color control, and structured multi-object generation.
Figure 14: Qualitative comparison on SD3.5-L. DA-DMD improves prompt adherence over DMD2 across diverse compositional challenges, including object counting, spatial placement, relative relations, color control, and structured multi-object generation.
Figure 15: Qualitative comparison on SANA. DA-DMD improves prompt adherence over DMD2 across diverse compositional challenges, including object counting, spatial placement, relative relations, color control, and structured multi-object generation.
Figure 16: Qualitative comparison on SANA-1.5. DA-DMD improves prompt adherence over DMD2 across diverse compositional challenges, including object counting, spatial placement, relative relations, color control, and structured multi-object generation.
Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student's distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. In multi-step distillation, although only the student's final output is delivered, existing methods train the fake model on intermediate states or across all denoising steps. Consequently, the student's update signal becomes entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after the student's final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student's final outputs and uses it to correct every denoising step. Our theoretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student's final output distribution matches the teacher's, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces single-step guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps. Experiments on SD3-Medium, FLUX.2-klein (4B and 9B), and Qwen-Image-2.1 demonstrate that eDMD consistently outperforms existing distribution matching methods for 4-step generation.
Tong Zhao, Xianfang Zeng, Yucheng Han +4
Zhejiang University · AGI Lab, Westlake University · StepFun
Distribution Matching Distillation (DMD) trains a few-step student from the difference between separately estimated target and student scores, so it must keep an auxiliary diffusion model fitted to the student's evolving distribution at extra memory and computation cost. We introduce DMAD, Distribution Matching as Adversarial Distillation, which recasts distribution matching as classification and learns the required log-density ratios directly. Two discriminator heads on a shared backbone distinguish real data and teacher samples from the student's, and linear losses on their logits train the student without auxiliary score fitting. We prove that at the discriminator optimum these losses recover the distribution-matching gradient underlying DMD, through the classical identity linking discriminator logits to log-density ratios. We further introduce gap-based reweighting, which adapts teacher supervision across noise levels from the real-data head's empirical logit gap between real and teacher samples. DMAD reaches a Fréchet Inception Distance (FID) of 1.04 with one-step generation on ImageNet-64x64, 14.47 with four-step SDXL on COCO-10K, and a VBench total score of 85.15 with four-step Wan2.1-T2V-14B, the best values among the compared few-step methods and the multi-step teachers. On MiniMax-H3-33B, our four-step student achieves overall human preference rates of 79.1% over DMD2 and 84.6% over rCM for joint audio-video generation, excluding ties. Our code, models and demos are available at https://yzmblog.github.io/projects/DMAD.
Zhengming Yu, Junkun Yuan, Haotian Yang +8
Texas A&M University · Intelligent Creation, ByteDance
Distribution Matching Distillation (DMD) distills multi-step diffusion models into few-step generators by minimizing reverse KL divergence. Despite its strong performance, DMD guides samples by local density alone, which introduces two limitations: teacher scores can be unreliable in regions of low teacher density, and the mode-seeking behavior of reverse KL can collapse the student onto a subset of the teacher modes. We first connect DMD to Wasserstein gradient flow, showing that its gradient follows from a fixed-point formulation with integral KL (IKL) as the energy functional. This perspective motivates augmenting IKL with an optimal transport term that addresses these limitations by guiding outlier samples toward the target and promoting mode coverage. We realize this term using Sinkhorn divergence estimated from finite batches. In practice, however, errors in the estimated Sinkhorn velocity can perturb in-domain samples for which IKL guidance is already accurate. To this end, we introduce sDMD, which adaptively activates the Sinkhorn correction based on discriminator outputs, focusing on outlier samples where IKL may be unreliable. With 2--4 sampling steps on FLUX.1-dev, Qwen-Image-20B, and Wan2.1-14B, sDMD outperforms distillation baselines on GenEval, DPG, WISE, and VBench while improving sample diversity over DMD2.