Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation
Authors: Youngjoong Kim, Duhoe Kim, Woosung Kim, Jaesik Park
Organizations: Department of Computer Science and Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
Consistency models have been proposed for fast generative modeling, achieving results competitive with diffusion and flow models. However, these methods exhibit instability and limited reproducibility when training from scratch, motivating subsequent work to explain and stabilize these issues. While these efforts have provided valuable insights, the explanations remain fragmented, and the theoretical relationships remain unclear. In this work, we provide a theoretical examination of consistency models by analyzing them from a flow map-based perspective. This joint analysis clarifies how training and convergence behavior can give rise to degenerate solutions. Building on these insights, we revisit self-distillation as a practical remedy for certain forms of suboptimal convergence. To avoid excessive gradient norms during its optimization, we reformulate its objective empirically, yielding improved Self-Distillation (iSD). We further extend iSD with training-time classifier-free guidance, yielding iSD-T, which improves performance and reproducibility in the studied settings. We demonstrate that iSD extends beyond image generation to diffusion-based policy learning, without reliance on pretrained diffusion models for initialization.
Figures & tables
Figure 1: From consistency training to improved Self-Distillation. Consistency training learns a mapping over conditional velocity, often suffering from training instability and reproducibility issues. Relaxing the time condition mitigates this instability, and self-distillation aligns a target with the marginal velocity field. However, directly applying self-distillation leads to unstable training. We therefore reformulate the objective and incorporate classifier-free guidance (iSD-T), further stabilizing training and improving reproducibility.
Model
Interpolant xt
Trajectory τt
Timestep
JVP
Objective
Distillation-based Methods
FMM-EMD ( Boffi et al., 2025a )
Any (Linear)
Φt(xt)
s<t
Exact
LED
AYF-EMD ( Sabour et al., 2025 )
Any (Linear)
Φt(xt)
s<t
Exact
LCD
sCD ( Lu and Song, 2025 )
Trigonometric
Φt(xt)
s=0
Exact
LCD
Consistency Training Methods
MeanFlow ( Geng et al., 2025a )
Linear
vt(xt∣x)
s<t
Exact
LCT
Table 1: Summary of recent consistency models interpreted under the flow map representation. Interpolants in parentheses denote settings realized in the cited work. Φt(xt) is the teacher network and Fθ(xt;t,s) is the training network. Our method is aligned with the learned marginal velocity Fθ(xt;t,t) and the relaxed time condition s<t , while reformulating Eulerian self-distillation (ESD) to reduce the gradient norm problem.
Figure 2: Toy experiments with a 5-layer MLP (batch size of 2048). LDT drives flow map training toward a suboptimal solution, while LED leads to a solution close to the ground truth.
Figure 3: (Left) Training results across batch sizes. The experiment follows Fig. 2 . Consistency training LCT exhibits drift toward degenerate distributions as the batch size B decreases. (Right) Estimating the optimality of flow maps using LED . Solid lines indicate a batch size of 2048, and dash-dot lines indicate 128.
Preconditioner
Preconditioner FID ↓
2-FID ↓
Multistep Baseline
1.21 (UCGM-S, 30-step)
2.52
LightningDiT
2.17 (Euler, 250-step)
9.59
In-house Flow Model
2.41 (UCGM-S, 30-step)
5.78
w/o Preconditioner
-
419.60
Reported Baseline
1.21 (UCGM-S, 30-step)
1.42
Table 2: Consistency training results under different preconditioners. Preconditioner FID denotes the FIDs of pretrained networks for given ODE solver and sampling-step pairs. 2-FID denotes two-step FIDs of consistency models initialized from the corresponding preconditioner (details in Sec. C.1 ).
Figure 4: Loss landscapes of four methods. The standard deviation σ and the number of spikes N serve as diagnostic proxies for training stability. iSD shows lower variance and fewer spikes under the common reference range. α and β denote the top-2 eigenvectors of the Hessian on ImageNet-1K with DiT-B/4. N denotes the number of samples outside each method’s own empirical 95% range. Values in parentheses report the number of samples exceeding the range defined by iSD, as a common reference (details in Sec. C.6 ).
Figure 5: Design choices. (a) FIDs over training steps. Solid lines indicate the JVP approximation, and dash-dot lines indicate the exact JVP. (b) FIDs of Post-CFG over guidance scales. (c) FIDs of Pre-CFG over training steps. Solid lines indicate trigonometric interpolation, and dash-dot lines indicate linear.
Case
FID ↓ ( ±σ )
IS ↑ ( ±σ )
Consistency Training
120.49 ( ± 3.71)
9.69 ( ± 0.47)
+ Joint training w/ LCFM
96.28 ( ± 15.9)
14.11 ( ± 4.15)
+ Relaxing to s<t
69.18 ( ± 2.84)
21.45 ( ± 1.21)
+ iSD (Eq. 16 )
114.56 ( ± 2.80)
11.04 ( ± 0.41)
+ JVP approximation
108.70 ( ± 2.83)
11.80 ( ± 0.36)
+ iSD-T (Eq. 19 , ω=5.0 )
26.91 ( ± 0.50)
74.70 ( ± 2.43)
Table 3: Ablation study on ImageNet with DiT-B/4. 2-step evaluation results with standard deviations σ .
Loss
Interp.
JVP
1.0
3.0
5.0
7.0
LCT
Linear
Exact
69.18
37.39
52.41
68.46
LiSD
Linear
Exact
114.56
74.17
78.20
88.48
LCT
Linear
Approx.
63.02
33.40
50.18
67.27
LiSD
Linear
Approx.
108.70
69.69
75.53
86.72
LiSD-T
Linear
Approx.
26.91
44.92
57.60
65.92
LiSD-T
Trig
Approx.
14.62
20.49
30.21
41.15
Table 4: FIDs across design choices (2-step). The numbers in the header denote Post-CFG scales.
Case
FID ↓ ( ±σ )
IS ↑ ( ±σ )
MeanFlow ( ω=3.0 )
11.48 ( ± 1.68)
167.60 ( ± 20.4)
Shortcut Model ( ω=1.5 )
87.52 ( ± 24.3)
14.94 ( ± 3.68)
FACM ( ω=1.75 )
25.52 ( ± 1.73)
65.42 ( ± 4.84)
iSD-T (Ours, ω=5.0 )
14.62 ( ± 0.06 )
184.53 ( ± 0.77)
Table 5: Comparison on ImageNet with DiT-B/4. 2-step results with standard deviations σ under a controlled setup: the backbone, data, optimization, and training budgets are fixed, while the only difference lies in the training objective. Each method is evaluated using the CFG scale reported as optimal in its work.
Table 11
Figure 6: 2-step results trained on ImageNet, iSD-T.
Model
NFE ( ↓ )
Push-T
Transport
Diffusion Policy ( Chi et al., 2023 )
100
0.95
1.00
1
0.12
0.00
Shortcut Model ( Frans et al., 2025 )
1
0.87
0.80
MeanFlow ( Geng et al., 2025a )
1
0.85
0.92
iSD (Ours)
2
0.92
1.00
1
0.86
0.94
Table 8: Success rates ( ↑ ) of robotic control tasks.
Figure 7: Simulation results of the iSD policy with 2-NFE on Transport (top) and Push-T (bottom) tasks.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Gradient norms of each method. The colored regions indicate 1 standard deviation. SD denotes Eulerian self-distillation (ESD), SD w/SG denotes the stop-gradient version of ESD introduced by Boffi et al. (2025b) , CT denotes consistency training, and iSD denotes our improved Self-Distillation.
Figure 9: Training curves of consistency training. (a) Training loss over training steps. (b) Two-step FIDs over training steps. (c) Rescaled Y-axis for the consistency model with a preconditioner.
Dataset
ImageNet-1K 256×256
CelebA-HQ 256×256
Preprocessor
SD-VAE ( Rombach et al., 2022 )
SD-VAE
VA-VAE ( Yao et al., 2025 )
SD-VAE
Input size
32×32×4
32×32×4
16×16×32
32×32×4
Condition
Class-conditional
Conditional
Conditional
Unconditional
Backbone
DiT-B/4
DiT-B/2
DiT-L/2
DiT-XL/2
DiT-XL/2 †
DiT-XL/1 †
DiT-B/2
Params (M)
131.33
131.33
459.42
676.78
682.02
681.77
131.33
Depth
12
12
24
28
28
28
12
Appendix
Table 9: Experimental Settings.
Figure 10: Gradient norm and loss curves of iSD-T compared to FID10K. FID and the flow matching loss decrease together over training, consistent with the role of velocity estimation in flow map training.
Table 10: Direct intervention on guiding velocity by interpolating marginal velocity ( γ=0 ) and conditional velocity ( γ=1 ). The marginal velocity guidance shows lower FID and lower loss variance. (Left) Toy experiments following Fig. 2 . We measure the optimality of flow maps using the proxy measure LED . (Right) Experiments in the ImageNet ablation setting. We report FIDs and the standard deviations σ of each loss landscape, following Fig. 4 .
Figure 11: Gradient norm and loss curves of consistency training, varying step size parameter d in Eq. 21 . Smaller d values are associated with lower loss values and better quantitative metrics in this setting.
Table 11: Ablation study for hyperparameters. We set ϵ=0.005 , η=0.01 , p=1.0 , and NFE=2 by default.
Figure 12: Variants of Pre-CFG. Dashed, dash-dot, and solid lines indicate iSD-U, iSD-C, and iSD-T, respectively. Each color represents a different Pre-CFG scale. Trigonometric interpolation and the JVP approximation are used by default on DiT-B/4. (a) FIDs of Pre-CFG variants over training steps. (b) FIDs of Post-CFG over guidance scales. (c) FIDs over multiplied guidance scales.
Post-CFG ω
0.0
0.5
1.0
1.5
3.0
5.0
7.0
10.0
FID
100.89
11.19
16.96
18.26
19.33
25.83
35.59
48.50
IS
13.53
164.48
234.85
229.35
181.94
116.25
85.75
65.36
Appendix
Table 12: FIDs and ISs across Post-CFG scales for DiT-XL/2 with iSD-T ( ω=5.0 , 700K steps).
Figure 13: FIDs on ImageNet. Solid lines indicate ω=5.0 , and dotted lines indicate the ω values in Tab. 5 .
Case
FID ↓
IS ↑
MeanFlow
13.12
220.9
Shortcut Model
128.7
6.57
FACM
26.85
85.05
Appendix
Table 13: Comparison at ω=5.0 .
Case
32
256
1024
Baseline
137.35
120.49
115.80
MeanFlow
31.63
11.48
8.84
iSD-T
31.47
14.62
13.10
iSD-T (DiT-B/2 † )
-
8.76
8.79
Appendix
Table 14: FIDs across batch sizes.
Figure 14: 4-step samples from iSD on CelebA-HQ 256×256 .
Figure 15: Training curves of self-distillation methods. The experimental settings follow the ImageNet ablation study. All listed methods adopt Pre-CFG with ω=5.0 .
Figure 16: Results of policy learning. The left two figures show average success rates (solid lines) and action mean-squared errors (dash-dot lines) over training steps. Numeric entries indicate the optimal values across training steps. The rightmost figure shows the simulation results of iSD with NFE=2.
Figure 17: Loss landscapes of four methods at 0K training steps. α and β denote the top-2 eigenvectors of the Hessian on ImageNet-1K with DiT-B/4. σ denotes the standard deviations of the landscape fields, and N the number of samples outside each method’s own empirical 95% range. Values in parentheses report the number of samples exceeding the 95% bound defined by iSD, as a common reference.
Figure 18: Loss landscapes at 200K training steps. Other settings are identical to those in Fig. 17 .
Table 15: Quantitative results (2-NFE). The numeric entries in the header denote training steps. Reported results are averaged over three random seeds unless otherwise specified; entries marked with an underline are averaged over five seeds. Std. denotes the standard deviation at 400K steps. Hyphens in Std. indicate cases evaluated with a single run, where standard deviation was not computed. Halted denotes cases where training diverged and was stopped earlier than the baseline.
Loss
Interp.
JVP
Arch.
Pre-CFG ω
100K
200K
300K
400K
Std.
Baseline
Linear
Exact
DiT-B/4
-
7.90
8.88
9.40
9.69
0.47
Joint training
Linear
Exact
DiT-B/4
-
10.95
13.01
14.32
14.11
4.15
LCT
Linear
Exact
DiT-B/4
-
8.76
11.34
19.78
21.45
1.21
LCT
Trig
Exact
DiT-B/4
-
11.77
15.04
16.79
18.11
0.98
LiSD
Linear
Exact
DiT-B/4
-
7.21
9.60
10.43
11.04
0.41
LiSD
Trig
Exact
DiT-B/4
-
9.37
12.02
13.53
14.75
0.70
Appendix
(b) Inception Score
Figure 19: Class-level samples generated by iSD-T with two-step sampling on ImageNet 256×256 .
Figure 20: Two-step samples from iSD-T on ImageNet 256×256 .
Consistency Models are generative models trained to map points along a diffusion trajectory to the same clean sample, enabling fast few-step generation.
The slow iterative sampling nature remains a major bottleneck for the practical deployment of diffusion and flow-based generative models. While consistency models (CMs) represent a state-of-the-art distillation-based approach for efficient generation, their large-scale application is still limited by two key issues: training instability and inflexible sampling. Existing methods seek to mitigate these problems through architectural adjustments or regularized objectives, yet overlook the critical reliance on trajectory selection. In this work, we first conduct an analysis on these two limitations: training instability originates from loss divergence induced by unstable self-supervised term, whereas sampling inflexibility arises from error accumulation. Based on these insights and analysis, we propose the Dual-End Consistency Model (DE-CM) that selects vital sub-trajectory clusters to achieve stable and effective training. DE-CM decomposes the PF-ODE trajectory and selects three critical sub-trajectories as optimization targets. Specifically, our approach leverages continuous-time CMs objectives to achieve few-step distillation and utilizes flow matching as a boundary regularizer to stabilize the training process. Furthermore, we propose a novel noise-to-noisy (N2N) mapping that can map noise to any point, thereby alleviating the error accumulation in the first step. Extensive experimental results show the effectiveness of our method: it achieves a state-of-the-art FID score of 1.70 in one-step generation on the ImageNet 256x256 dataset, outperforming existing CM-based one-step approaches.
Linwei Dong, Ruoyu Guo, Ge Bai +3
1Zhejiang University · 2Bytedance Inc. · 3Zhejiang Lab
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256×256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.
Yuxiang Fu, Qi Yan, Zike Wu +4
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
Continuous-time consistency models are an attractive route to few-step generation. They remove the discretization error inherent to the timestep grid that discrete-time consistency models depend on. Until recently, however, training them was unstable. Lu & Song (2025) address this with the TrigFlow formulation and a coordinated set of architectural and training-objective changes. We conduct an independent empirical study of this framework on CIFAR-10 and ImageNet-64. No open-weight TrigFlow teachers exist at these scales, so we train them from scratch and then verify the consistency distillation (sCD) and consistency training (sCT) behavior of students initialized from these teachers. On CIFAR-10, our students reproduce the reported FID at one and two steps. On ImageNet-64, they fall short, a gap that appears related to our teacher and to architectural choices the paper leaves unspecified. We also report a parameter-count gap of about 5% between the model size stated in the paper and a faithful re-implementation, which we attribute to the scale-and-shift conditioning projections in Adaptive Double Normalization. We then propose Multistep sCD (MS-sCD), a continuous-time, segment-conditioned multistep generalization of sCD. The segment-conditioning idea is borrowed from the discrete-time multistep consistency distillation of Heek et al. (2024); we build on their preliminary continuous-time experiments within the TrigFlow/sCM framework. The rest of the algorithm follows sCD. MS-sCD replaces sCM's two-step intermediate time with a fixed segment-boundary schedule and recovers sCD at M = 1. We report FID at M = 2, 4, 8 and compare against the discrete-time MSCD baseline. Code, model weights, and training scripts are available on GitHub (Mathew et al., 2026).
Vinay Saji Mathew, Soundar R. Kumara, Gretta D Kellogg +1
The Pennsylvania State University · State University of New York at Buffalo