Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher's score and an online critic's estimate of the student distribution's score. We investigate what the critic needs, in capacity and in information, to provide a reliable training signal. On the capacity side, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher, a strategy we call joint scaling, prevents collapse in our experiments and allows the student to benefit from larger teachers. On the information side, rather than forcing the critic to model a step-agnostic mixture of student outputs, we introduce a step-aware critic (STAC) that conditions on the student's generation step to estimate step-specific scores. Together, joint scaling and STAC consistently improve few-step distillation across class-conditional ImageNet generation, text-to-image generation with SDXL and FLUX.2, and text-to-video generation with Wan2.1.
Figures & tables
Figure 1: Few-step students distilled with a critic as large as the teacher. Top, samples of a 4-step FLUX.2 [klein] 4B student distilled with a 32B teacher and a 32B critic with STAC. Bottom, three frames each of six videos from a 4-step Wan2.1 1.3B student distilled with a 14B teacher and a 14B critic with STAC. (Settings: FLUX.2 1024×1024 , Wan2.1 480p.)
Figure 2: Overview. We study (a) step conditioning and (b) joint teacher–critic scaling. In (b), T and F denote teacher/critic capacity: T>F can be unstable, T<F inefficient, and T=F favorable here.
Figure 3: The joint scaling phenomenon. Starting from an S-sized teacher, critic, and student and fixing the student size, each panel scales (a, left) the teacher, (b, middle) the critic, or (c, right) both along the ladder, with the horizontal axis the parameter count of the scaled network. Scaling only the teacher initially improves FID but eventually degrades it and causes collapse at H; scaling only the critic yields marginal gains; scaling both improves the student at every size. (Settings: 1-step S student, ImageNet 256×256 , w/o CFG; best FID with the per-cell learning-rate selection of Appendix B.2 .)
FID ↓
T
F
lowest
100k
200k
IS ↑
Precision ↑
Recall ↑
B
B
16.82
18.78
16.82
99.85 ± 2.36
0.762
0.290
XL
B
18.56
21.91
41.75
107.19 ± 1.97
0.735
0.224
B
XL
16.61
18.36
19.12
105.25 ± 2.38
0.698
0.333
XL
XL
13.50
14.72
13.50
110.14 ± 2.49
0.760
0.388
Table 1: Training dynamics. The four runs of Figure 5 a, a 1-step B student under B and XL teachers ( T ) and critics ( F ) with one recipe: the lowest FID over training, the FID at 100k and 200k updates, and IS, precision, and recall at the lowest-FID checkpoint. XL/XL leads on every metric except precision, where it trails the best by 0.002, and only the two matched pairs reach their lowest FID at the end of training. (Settings: ImageNet 256×256 , w/o CFG, default learning rates, 200k updates.)
Figure 4: Per-update training compute versus FID. Best FID versus TFLOPs per student update for every non-collapsed teacher/critic pair of the S-student grids at (a) 1 and (b) 4 steps. Matched pairs are labeled by size and joined in size order; the nonlinear FID axis expands the low-FID region. At comparable per-update compute, matched pairs sit at or near the lowest FID, while most mismatched pairs lie above them. (Settings: ImageNet 256×256 , w/o CFG; learning rates as in Figure 10 ; FLOPs from Table 6 .)
Figure 5: Scaling the teacher and critic on ImageNet. Every panel varies the teacher and critic sizes while the student is fixed. (a) FID over training for a 1-step B student under four teacher/critic pairs (B/B, XL/B, B/XL, XL/XL); the mismatched XL/B pair degrades only late, so (a) runs to 200k updates to expose it. (b, c) A 1-step S student over a grid of L, XL, and H teachers and S-to-H critics: (b) reports the best FID over training, and (c) the FID at a fixed 100k-update budget. Red boxes mark the diagonal , matched teacher/critic pairs. For both students, matched pairs give the lowest FID, while mismatched pairs often degrade and sometimes collapse (best FID of 100 or higher). (Settings: ImageNet 256×256 , w/o CFG; (a) default learning rates; (b, c) the best learning-rate pair per cell by the lowest FID over training.)
Figure 6: Generation quality varies sharply across steps. Clean predictions of a 4-step SDXL student along one generation path with a shared prompt and seed. Colored boxes and insets magnify the regions discussed below. The first step already fixes the overall color and layout but still carries significant artifacts; the second step refines the image substantially, yet artifacts remain in detailed regions such as the face and the wheel; the third step retains only small residual artifacts around the mouth, which the last step removes.
T / F
FID ↓
IS ↑
Prec. ↑
Rec. ↑
B / B
13.91
116.81 ± 1.98
0.792
0.334
+ STAC
11.77
127.90 ± 1.84
0.815
0.340
XL / XL
8.08
161.01 ± 3.75
0.739
0.452
+ STAC
6.90
165.16 ± 3.13
0.731
0.470
Table 2: Cumulative results and step-input ablations on ImageNet. Left: a 4-step B student under B/B and XL/XL teacher–critic pairs, without and with STAC (“+ STAC”); joint scaling and STAC each lower FID, and together they give the best student. Right: comparison of the XL/XL student with STAC against three controls that replace its learned in-context step tokens, namely const tokens shared by all steps, step-specific but random and untrained frozen tokens , and an AdaLN branch that injects the step through adaptive normalization. Relative to the XL/XL baseline without STAC (FID 8.08), all three controls recover a smaller FID gain than STAC (6.90). Every run shares the recipe, and IS is given with its standard deviation over splits. (Settings: ImageNet 256×256 , w/o CFG, default learning rates, FID at 100k updates.)
Figure 7: Training dynamics of STAC. FID during training of a 4-step student without STAC (dashed) and with STAC (solid). With STAC, FID improves steadily after 40k updates. (Settings: XL student, teacher, and critic, ImageNet 256×256 , CFG =1.3 .)
Few-step: ours, 4B student, 4 steps
Multi-step: base model, 50 steps
T
F
STAC
HPSv3 ↑
DPG-Bench ↑
Model
HPSv3 ↑
DPG-Bench ↑
4B
4B
w/o
10.58
81.58
4B
10.51
82.88
9B
4B
w/o
10.32
82.08
9B
9B
w/o
10.92
83.70
9B
9B
w/
11.86
84.22
9B
10.58
85.28
32B
32B
w/o
12.10
84.98
Table 3: FLUX.2 benchmark at 1024×1024 . Left: our 4B students with 4-step sampling, distilled with different teacher ( T ) and critic ( F ) sizes. The 4B and 9B models used as teachers or critics come from FLUX.2 [klein] base; the 32B models come from FLUX.2 [dev]. Right: the official base model of the teacher’s size, sampled with 50 steps and CFG 4.0. Details in Appendix B.4 .
Method
FID ↓
CLIP-S ↑
ImageReward ↑
HPSv3 ↑
LCM-LoRA ( Luo et al., 2023a )
22.27
31.71
39.56
6.45
SDXL-Turbo ( Sauer et al., 2023 )
27.27
32.16
46.09
9.09
SDXL-Lightning ( Lin et al., 2024 )
24.49
32.31
57.48
9.48
Flash ( Chadebec et al., 2024 )
22.96
31.84
19.04
6.49
PCM ( Wang et al., 2024 )
24.13
32.52
64.73
9.46
DMD2, official ( Yin et al., 2024a )
18.95
33.14
71.01
9.64
Table 4: Comparison with distilled SDXL systems. SDXL systems with 4-step sampling on 10k COCO prompts at 512×512 . The first seven rows are the values reported by Liu et al. (2025) , and our rows follow the same protocol. CLIP-S and ImageReward are multiplied by 100. Continuing DMD2 training with STAC improves clearly over the official checkpoint on every metric, whereas the same continuation without STAC falls below it. Detailed settings in Appendix B.3 .
Figure 8: Controlled comparison of STAC on Wan2.1. VBench scores during training for bidirectional DMD (top) and autoregressive Self Forcing (bottom), without and with STAC. STAC leads in VBench total at every evaluation in both regimes. (Settings: 1.3B teacher, student, and critic; ODE initialization from the CausVid repository ( Yin et al., 2025 ) ; 4 steps, 480p, 2k iterations; bidirectional DMD uses no backward simulation, while the autoregressive runs use an online Self Forcing rollout.)
Configuration
Total ↑
Quality ↑
Semantic ↑
Dynamic ↑
CausVid (reproduced)
82.76
83.49
79.85
–
Self Forcing (our eval.)
83.87
84.59
81.00
50.12
Causal Forcing ( Zhu et al., 2026 )
T=14 B, F=1.3 B (Default)
83.94
84.73
80.79
52.06
T=1.3 B, F=1.3 B
83.96
85.14
79.26
81.39
T=14 B, F=14 B
84.26
85.04
81.16
57.52
Table 5: Critic size and STAC for 1.3B Wan2.1 students. The first rows are Self Forcing’s CausVid reproduction and our Self Forcing evaluation ( Yin et al., 2025 ; Huang et al., 2025 ) . Controlled rows share a Causal Forcing ODE initialization and Self Forcing rollout; the default is 14B/1.3B, and the last adds STAC to 14B/14B. A 14B critic improves all metrics (Dynamic: 52.06 to 57.52); STAC improves them further. (4 steps, 480p VBench; Appendix B.5 .)
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
GFLOPs
Model
Depth
Width
Heads
Head dim.
Params. (M)
Fwd.
Bwd.
FID ↓
S
12
384
6
64
33.0
10.9
21.8
50.99
S-B
12
512
8
64
58.3
19.4
38.8
40.64
B
12
768
12
64
130.5
43.6
87.2
29.50
M
18
896
14
64
264.2
89.0
178.0
20.66
L
24
1024
16
64
458.1
155.0
309.9
15.91
Appendix
Table 6: The teacher and critic ladder. Architecture, forward and backward GFLOPs per sample, and teacher FID of the seven SiT sizes after 800k pretraining updates. Teacher FID falls steadily from S to H. (Settings: ImageNet 256×256 , w/o CFG; teacher sampling as in Table 7 .)
Setting
Value
Training objective
Flow Matching ( Lipman et al., 2023 )
Optimizer
AdamW; β=(0.9,0.999) ; weight decay 0
Learning rate
10−4 , constant; no warmup or schedule
Batch size
256
Training length
800k updates
EMA decay
0.9999
Appendix
Table 7: Hyperparameter choices of ImageNet teacher pretraining.
FID ↓
t -sampler
ηG
ηF
S
B
L
Uniform
10−6
10−6
51.16
30.77
12.28
Uniform
8×10−6
10−6
37.13
19.11
8.91
Logit-normal
8×10−6
10−6
33.96
16.82
8.28
Appendix
Table 8: The pure DMD baseline. FID of 1-step students after 200k updates at three sizes, with student, critic, and teacher of equal size, for the uniform and logit-normal samplers of the score-evaluation level and two student learning rates. Each change lowers FID at every size, and the last row is our default ImageNet recipe. (Settings: ImageNet 256×256 , w/o CFG, one critic update per student update.)
Setting
Value
Objective
Equation 5 ; no auxiliary regression or GAN loss
Initialization
Pretrained 800k EMA checkpoint of each network’s own size
Batch size
64
Optimizer
AdamW with β=(0.9,0.999)
Learning rates ( ηG / ηF )
8×10−6 / 10−6 unless stated otherwise
Weight decay
0.01
Appendix
Table 9: Hyperparameter choices of ImageNet distillation experiments. Sizes, learning rates, and budgets of individual comparisons are given with each experiment.
Setting
Pure DMD ablation
Main results
Step schedule
{999,749,499,249}
Step input to critic
See Table F.3
4 in-context tokens (w/ STAC only)
Learning rates
Student and critic 5×10−7 ; step tokens 10−4
Optimizer
AdamW, β=(0.9,0.999) ; gradient clipping at norm 10
Batch size
128
Critic updates per student update
5
Appendix
Table 10: Hyperparameter choices of SDXL experiments. Our core comparison table (Table 4 ) uses the configurations on the right, while ablations in Table F.3 use a more lightweight setting on the left.
Setting
Value
Step schedule
{1000,948,858,668} (time-shifted from {1000,750,500,250} )
Table 12: Hyperparameter choices of the main Wan2.1 experiments (Table 5 ).
Figure 9: Joint scaling and STAC on a two-dimensional example. Samples of a 2-step, depth-3 student at each sampling step (rows), with energy distance (ED) to the target. The first column shows the single-roll target and an undistilled depth-3 teacher sampled with 20 Euler steps. The next three columns vary the teacher ( T ) and critic ( F ) depths, and the last adds STAC to the depth-6 pair. Deepening only the teacher makes both steps worse, deepening the critic as well reverses this, and STAC brings both steps closer to the single roll. (Settings: 4k DMD updates, five critic updates per student update; Table 13 .)
Setting
Value
MLP width
256
Teacher training steps
12k with batch size 1,024
DMD training steps
4k with batch size 256
Critic updates per student update
5
Optimizer
Adam with β=(0.9,0.999)
Teacher learning rate
10−3 with cosine decay
Appendix
Table 13: Hyperparameter choices of the two-dimensional toy experiment.
Figure 10: The expanded teacher–critic scaling grid. Best FID per teacher/critic pair for S and B students at 1 and 4 steps. Red boxes mark the matched pairs, and hatched cells mark collapsed runs, whose best FID is 100 or higher under all nine learning-rate pairs. Every collapse occurs with a critic smaller than its teacher. (Settings: ImageNet 256×256 , w/o CFG, 200k updates; learning-rate selection as in Figure 5 b.)
Teacher / critic
S / S
H / S
H / S
H / H
Critic learning rate
searched
searched
16× default
searched
S student, 1 step
33.96
247.18
52.53
21.41
S student, 4 steps
29.91
281.41
42.89
14.65
Appendix
Table 14: Forcing a small critic with an extreme learning rate. Best FID of 1- and 4-step S students under the H teacher with an S critic. We compare the searched learning rates against an aggressively scaled critic learning rate ( 16× default), alongside the matched S/S and H/H baselines. Although the larger learning rate prevents the collapse, the generation quality stays far behind that of the matched H/H pair. (Settings: ImageNet 256×256 , w/o CFG, best FID over training; 16× runs use (ηG,ηF)=(8×10−6,1.6×10−5) ; other cells use the best of the nine searched pairs, as in Figure 10 .)
Figure 11: Guidance for DMD students and their teachers. FID against guidance scale for (a) 1-step B students distilled under CFG, (b) their teachers sampled with CFG, and (c) students distilled with a weak B model as the autoguidance reference. The students’ best CFG scale, about 1.3, lies below the teachers’ 1.5, and autoguidance does not improve on CFG. (Settings: ImageNet 256×256 , F=T for M, XL, and H; students report their best FID over training.)
G
Steps
S
S-B
B
M
L
XL
H
S
1
24.07
20.75
18.55
16.98
19.10
16.19
20.17
S
2
18.17
15.85
12.83
10.92
11.69
10.34
11.91
S
4
17.47
14.98
11.09
9.72
9.95
8.64
10.54
B
4
13.81
11.31
7.05
6.40
6.03
5.52
5.76
Appendix
Table 15: Joint scaling with classifier-free guidance. Best FID for each teacher (columns) with the critic fixed at XL. Rows vary the student and its step count. Every row is best at the XL teacher, the one matching the critic, and worse again at the larger H teacher. (Settings: ImageNet 256×256 , CFG =1.2 ; other settings follow Appendix B.2 .)
Figure 12: Teacher scaling under velocity KD and MeanFlow distillation. Best FID against teacher size for (a) an S student regressed onto each larger teacher’s velocity (the S teacher was not evaluated) and (b) S and B students distilled with MeanFlow at 1 and 4 steps. No teacher collapses these students, and the gain from a larger teacher shrinks as it grows. (Settings: ImageNet 256×256 ; KD: learning rate 10−4 , 200k updates; MFD: official MeanFlow recipe with default guidance coefficients, 200k updates for S and 60k for B students.)
Figure 13: More critic updates for a small critic. FID during training with the student-size critic at one and five updates per student update, and with an XL critic at one update, for (a) a B student with an L teacher, (b) a B student with an XL teacher, and (c) an S student with an XL teacher. Five updates speed up early training and avoid the collapse in (a), though the student-size critic stays at a higher FID than the single-update XL critic. (Settings: ImageNet 256×256 , 1-step students, w/o CFG.)
Figure 14: Critic warm-up. (a) FID of a 1-step S student under an H teacher with an S critic. The S critic is either trained concurrently from the start or warmed up for 10k/40k updates before student training begins. Both warm-ups fail to prevent collapse compared to the matched H critic. (b) Validation loss of an S critic fitted to a frozen student. (Settings: ImageNet 256×256 , w/o CFG.)
Critic
Teacher
B
XL
XL, 7M
B
17.94
16.61
17.64
XL
18.56
13.71
11.49
XL, 7M
35.96
12.01
11.90
Appendix
Table 16: Teachers and critics from different checkpoints. Best FID of a 1-step B student. Teachers (rows) and critics (columns) come from our B and XL models pretrained for 800k updates, alongside the highly converged XL model from Ma et al. (2024) trained for 7M updates (XL, 7M). Shaded cells denote shared initial weights. Regardless of the teacher’s training duration, matching the critic’s architectural size (XL) gives stable and strong performance, whereas a B critic degrades under both XL teachers. (Settings: ImageNet 256×256 , w/o CFG, default learning rates, 100k updates.)
Trainable
Critic learning rate
Critic
params (M)
3×10−5
10−4
3×10−4
LoRA on XL, r=1
0.74
14.61
13.78
14.99
LoRA on XL, r=4
2.97
13.66
14.97
16.34
LoRA on XL, r=16
11.87
14.82
16.92
17.28
LoRA on XL, r=64
47.48
13.84
17.68
17.75
Full B (student size)
130.5
18.56
Appendix
Table 17: Low-rank critics on the frozen teacher. Best FID of a 1-step B student under an XL teacher. LoRA critics adapt the frozen XL teacher, and the full-parameter critics are the corresponding cells of Figure 10 b, with the learning rates selected per cell. The best LoRA result is underlined. Even at rank 1, every LoRA critic beats the full B critic, though none reaches the full XL critic. (Settings: ImageNet 256×256 , w/o CFG; LoRA runs: ηG=1.2×10−5 , 100k updates, batch size 64, scaling α/r with α=8 , zero-initialized up-projection.)
Figure 15: Step information in re-noised predictions. Lower bound on the mutual information I(K;Xt) from a classifier predicting the step index of a 4-step B student, evaluated at (a) 30k and (b) 100k updates with online weights, and (c) at 100k updates with EMA weights. The step is readable at low and moderate score-evaluation levels, and STAC lowers this classifier-derived bound at nearly every level. (Settings: ImageNet 256×256 , XL teacher and critic, w/o CFG; 10k fixed noise–class pairs, 8k/2k training/held-out split.)
Figure 16: Loss variability with and without STAC. Coefficients of variation of (a) the critic loss and (b) the DM loss during the training of a 4-step student. STAC (solid) has a lower CV than the step-agnostic baseline (dashed) at every evaluation after the first. (Settings: XL student, teacher, and critic, ImageNet 256×256 , CFG =1.3 .)
Conditioning
Channel
Tokens
FID ↓
CLIP-S ↑
IS ↑
None
None
0
34.71
32.65
34.78
Step index
AdaGN
0
34.41
32.65
36.25
Step tokens
Cross-attention
4
32.73
32.64
35.29
Step tokens
Cross-attention
8
33.72
32.59
37.15
Step tokens
Cross-attention
16
34.01
32.60
36.51
Appendix
Table 18: Step conditioning of the SDXL critic. Four cross-attention step tokens yield the best FID, outperforming both AdaGN and larger token counts (8 or 16). (Settings: SDXL student and critic, 4 steps, 8k updates, 10k COCO prompts; pure DMD without GAN loss.)
System
Added parameters
Iteration time
Peak memory
SDXL
+0.18%
≈0%
+0.70%
Wan2.1, autoregressive
+0.19%
≈0%
+0.15%
Appendix
Table 19: Cost analysis of STAC on SDXL and Wan2.1. Changes relative to each system’s baseline without STAC, with the inference architecture unchanged. STAC adds a negligible amount of memory and iteration time.
Figure 17: Uncurated ImageNet 256 × 256 samples of 1-step B students. Each 3×3 grid shows nine samples of one selected class. The first column is the B teacher without guidance, with its FID from Table 6 , and the other columns are the students of Figure 5 a at 200k updates, labeled by their teacher/critic pair, their FID (Table 1 ), and the change from B/B. Scaling both networks (XL/XL) gives more coherent objects than B/B, and scaling only the critic (B/XL) changes little, whereas scaling only the teacher (XL/B) produces near-duplicate samples within each class, indicating low diversity. (Settings: w/o CFG.)
Figure 20: Wan2.1 samples of 4-step 1.3B students across teacher/critic pairs. Each row shows four frames of one video, and the three rows of a block share the prompt given below them. Big and small denote the 14B and 1.3B models. Frame change is the mean absolute luminance difference between consecutive shown frames, expressed as a percentage of the intensity range; higher values indicate greater temporal change, which may reflect either motion or artifacts. Red boxes mark artifacts. Relative to the big-teacher/small-critic pair, matching the critic to the big teacher recovers part of the temporal change and yields more plausible videos. (Settings: autoregressive Wan2.1-T2V-1.3B students with the Self Forcing rollout, 4 steps, 480p; teacher/critic pairs of Table 5 .)