Forward-Value Discretization: Faster Stochastic Diffusion Sampling Without High-Order Approximation
Authors: Yuchen Jiao, Na Li, Changxiao Cai, Yuting Wei, Gen Li
Organizations: Department of Statistics and Data Science, Chinese University of Hong Kong, Hong Kong · Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA · Department of Statistics and Data Science, the Wharton School, University of Pennsylvania, Philadelphia, USA
Stochastic diffusion samplers often require substantially more neural function evaluations (NFEs) than deterministic samplers to achieve comparable sample quality. Although higher-order discretization accelerates deterministic sampling, its benefits for stochastic sampling remains unclear. We establish matching upper and worst-case lower bounds showing that stochastic samplers with higher-order approximations remain first-order convergent, even with exact score functions. Thus, increasing the approximation order does not generally improve the convergence order. Motivated by this limitation, we introduce forward-value discretization, a training-free framework that evaluates the data/noise prediction model at a lookahead estimate constructed from previous predictions. For deterministic sampling, the resulting Forward DPMSolver (F-DPMSolver), improves sample quality at low NFE budgets through its DDIM-based variant, despite remaining first-order convergent. For stochastic sampling, we develop Forward DDPM (F-DDPM), which evaluates the model at a cleaner lookahead point adjusted for subsequent noise injection. This estimate reuses both the previous model prediction and the previously injected noise. Both variants admit first-order convergence guarantees with one model evaluation per iteration. Across CIFAR-10, ImageNet, FFHQ, and LSUN, F-DDPM consistently achieves lower FID than the evaluated stochastic baselines at matched NFE budgets. With 1.5 times as many NFEs as deterministic baselines, it achieves competitive sample quality, narrowing the practical efficiency gap between stochastic and deterministic sampling. Code is available at https://github.com/Na-Li66/F-DDPM.
Figures & tables
Figure 1: Stable Diffusion samples at different NFEs for the cat and rodeo prompts described in the text. In these examples, SDE-DPMSolver++1 and SDE-DPMSolver++2 depict the cat and human figures more clearly than DDIM and DPMSolver-3.
Figure 2: FID comparison of DDIM, DPMSolver-2, and a forward-value discretization on CIFAR-10 (EDM-uncond) and ImageNet-64 (EDM2-S), versus the number of outer iterations M . The forward-value discretization uses additional model evaluations to estimate endpoints. FID is computed using 50K generated samples.
Model
NFE
Higher-order Algorithms
First-order Algorithms
DPMSolver-2
DPMSolver-3
UniPC-3
DDIM
F-DDIM ( ours )
EDM -uncond
4
43.55
33.77
106.07
66.72
25.01
5
27.86
18.20
50.35
49.62
16.04
6
17.70
10.68
50.09
35.58
9.47
8
9.74
5.60
6.54
22.30
4.91
10
6.52
3.88
2.73
15.67
3.46
Table 1: FID on unconditional and conditional CIFAR-10 with EDM.
Model
NFE
Higher-order Algorithms
First-order Algorithms
DPMSolver-2
DPMSolver-3
UniPC-3
DDIM
F-DDIM ( ours )
EDM2 -XS
4
66.23
59.35
200.75
87.81
50.96
5
38.02
28.06
87.59
61.62
22.69
6
28.24
16.11
23.93
53.89
14.33
8
12.19
7.01
7.48
31.17
6.18
10
7.45
4.93
4.72
20.97
4.57
Table 2: FID on ImageNet-512 with EDM2-XS/XXL.
Model
Deter- ministic NFE
Stochastic Algorithms (1.5 × NFE)
Deterministic Algorithms
SDE DPMSolver++1
SDE DPMSolver++2
F-DDPM ( ours )
DDIM
DPMSolver-3
F-DDIM ( ours )
CIFAR10 -uncond
4
77.58
37.24
21.85
67.03
33.77
25.02
6
55.52
17.89
8.63
35.89
10.69
9.47
8
41.79
10.55
4.62
22.46
5.60
4.92
10
32.64
7.07
3.34
15.85
3.88
3.46
CIFAR10 -cond
4
56.21
27.83
15.11
47.27
23.99
18.44
Table 3: FID on unconditional and conditional CIFAR-10.
Model
Deter- ministic NFE
Stochastic Algorithms (1.5 × NFE)
Deterministic Algorithms
SDE DPMSolver++1
SDE DPMSolver++2
F-DDPM ( ours )
DDIM
DPMSolver-3
F-DDIM ( ours )
ImageNet 64-S
4
49.04
25.95
24.01
39.37
21.49
20.55
6
34.50
13.65
8.63
20.65
6.96
6.67
8
25.77
8.68
4.38
12.56
3.74
3.50
10
19.94
6.16
3.06
8.76
2.70
2.56
ImageNet 64-L
4
53.86
27.89
25.63
43.86
23.66
22.35
Table 4: FID on ImageNet-64 with EDM2-S/L.
Model
Deter- ministic NFE
Stochastic Algorithms (1.5 × NFE)
Deterministic Algorithms
SDE DPMSolver++1
SDE DPMSolver++2
F-DDPM ( ours )
DDIM
DPMSolver-3
F-DDIM ( ours )
ImageNet 512-XS
4
102.69
45.22
37.86
87.78
59.34
50.96
6
65.38
15.60
10.86
53.88
16.11
14.33
8
44.63
7.84
5.75
31.16
7.01
6.18
10
31.97
5.55
4.47
20.97
4.93
4.57
ImageNet 512-XXL
4
100.40
40.10
36.71
91.45
60.11
52.32
Table 5: FID on ImageNet-512 with EDM2-XS/XXL.
Model
Deter- ministic NFE
Stochastic Algorithms (1.5 × NFE)
Deterministic Algorithms
SDE DPMSolver++1
SDE DPMSolver++2
F-DDPM ( ours )
DDIM
DPMSolver-3
F-DDIM ( ours )
LSUN
4
89.39
15.40
15.09
70.55
10.13
12.17
6
41.82
12.32
5.17
26.15
9.30
5.56
8
23.88
13.23
4.50
13.70
9.24
4.92
10
15.54
13.86
4.47
9.00
8.38
4.65
FFHQ
4
114.77
23.55
22.57
80.19
15.65
15.78
Table 6: FID on LSUN bedrooms and FFHQ with latent diffusion models.
Figure 3: Qualitative comparison of F-DDPM, SDE-DPMSolver++1, and SDE-DPMSolver++2 on ImageNet-64 using EDM2-S and EDM2-L. All methods use the NFE indicated above each column.
Figure 4: Qualitative comparison of F-DDPM, SDE-DPMSolver++1, and SDE-DPMSolver++2 on FFHQ using a pre-trained latent diffusion model. All methods use the NFE indicated above each column.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Model
NFE
Higher-order Algorithms
First-order Algorithms
DPMSolver-2
DPMSolver-3
UniPC-3
DDIM
F-DDIM ( ours )
EDM2 -S
4
27.00
21.50
54.41
39.38
20.55
5
16.14
11.16
28.88
28.11
10.70
6
10.66
6.96
16.22
20.65
6.67
8
5.83
3.74
6.13
12.56
3.50
10
3.98
2.71
3.05
8.77
2.56
Appendix
Table 7: FID on ImageNet-64 with EDM2-S/L.
Dataset
NFE
Higher-order Algorithms
First-order Algorithms
DPMSolver-2
DPMSolver-3
UniPC-3
DDIM
F-DDIM ( ours )
FFHQ
4
31.26
15.67
77.43
80.30
15.79
5
16.33
11.28
21.93
58.42
9.75
6
10.47
11.53
9.96
43.75
7.89
8
7.19
12.08
7.37
27.05
6.80
10
6.58
11.62
6.82
18.81
6.46
Appendix
Table 8: FID on FFHQ and LSUN bedrooms with latent diffusion models.
Figure 5: Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained EDM2 with S and L size on ImageNet64 dataset.
Figure 6: Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained EDM2 with XS and XXL size on ImageNet512 dataset.
Figure 7: Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained unconditional EDM on CIFAR10 dataset.
Figure 8: Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained conditional EDM on CIFAR10 dataset.
Figure 9: Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained latent diffusion model on FFHQ dataset.
Figure 10: Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained latent diffusion model on LSUN bedroom dataset.
NFE
EDM-uncond
EDM-cond
F-DDIM
F-DPMSolver-2
F-DDIM
F-DPMSolver-2
4
25.01
22.14
18.44
17.21
5
16.04
14.97
12.29
11.37
6
9.47
8.44
7.57
6.70
8
4.91
4.13
4.32
3.57
10
3.46
2.79
3.18
2.47
Appendix
Table 9: FID comparison of F-DDIM and F-DPMSolver-2 on CIFAR-10.
NFE
EDM2-XS
EDM2-XXL
F-DDIM
F-DPMSolver-2
F-DDIM
F-DPMSolver-2
4
50.96
47.63
52.32
47.06
5
22.69
20.30
23.56
18.97
6
14.33
11.93
14.04
10.88
8
6.18
6.09
5.04
4.24
10
4.57
4.90
3.06
2.89
Appendix
Table 10: FID comparison of F-DDIM and F-DPMSolver-2 on ImageNet-512.
Figure 11: Qualitative comparison of F-DDIM and F-DPMSolver-2 on CIFAR-10 using unconditional and conditional EDM models.
Figure 12: Qualitative comparison of F-DDIM and F-DPMSolver-2 on ImageNet-512 using EDM2-XS and EDM2-XXL.
NFE
DPM Solver-2
DPM Solver-3
UniPC -3
DDIM
F-DDIM ( ours )
10
6.58
3.60
3.15
21.48
3.06
12
4.44
2.80
2.53
15.02
2.48
15
3.19
2.39
2.24
9.76
2.23
20
2.52
2.16
2.11
5.98
2.11
Appendix
Table 11: FID at larger NFE budgets on ImageNet-512 with EDM2-XXL.
NFE
4
5
6
8
10
DDIM
4×1.18
5×1.12
6×1.32
8×1.04
10×1.00
DPMSolver-2
4×1.17
5×1.10
6×1.07
8×1.02
10×1.00
DPMSolver-3
4×1.59
5×1.47
6×1.16
8×1.09
10×1.05
UniPC-3
4×1.18
5×1.16
6×1.38
8×1.08
10×1.05
F-DDIM (ours)
4×1.15
5×1.08
6×1.06
8×1.03
10×0.99
F-DPMSolver-2 (ours)
4×1.13
5×1.07
6×1.04
8×1.01
10×0.99
Appendix
Table 12: Normalized runtime on ImageNet-64 with EDM2-S. DDIM with NFE =10 is assigned a value of 10 .
Figure 13: Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained unconditional EDM on CIFAR10 dataset.
Figure 14: Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained conditional EDM on CIFAR10 dataset.
Figure 15: Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained EDM2 with XS and XXL size on ImageNet512 dataset.
Figure 16: Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained latent diffusion model on LSUN bedroom dataset.
Diffusion models rely on numerical solvers requiring time-discretization, which has a large influence on the tradeoff between sampling cost and quality. However, the computational difficulty of the reverse process varies along the sampling trajectory and across data distributions, making the choice of discretization important. We adapt proportional-integral (PI) step-size control to diffusion, using our diffusion noise-normalised error estimator. Unlike existing adaptive methods in diffusion that respond only to the current error, the PI solver also incorporates the previous error, yielding smoother step adaptation. We further show that these per-sample trajectories exhibit shared structure and can be aggregated into a fixed schedule that retains much of the benefit of adaptive sampling. We evaluate both approaches on natural-image and language datasets, in terms of quality, measured by FID at a matched number of neural network evaluations (NFE), comparing them with widely used stochastic solvers and schedules. For images, our fixed discretization outperforms the commonly used EDM schedule in terms of sample quality when used with the stochastic Heun sampler, and with the EDM-churn sampler at low NFE. Additionally, our PI adaptive solver obtains better FID than most stochastic and adaptive baselines, although it does not beat the EDM-churn sampler at low NFE. Moreover, we find our solver outperforms both the EDM and the entropy schedule on language diffusion at low-to-medium NFE in terms of perplexity, with the drawback of lower token entropy. Lastly, we find that the benefit of per-sample adaptivity is problem-dependent. It is highly beneficial in 1D toy examples, while only marginal for image and language data, where the average schedule sometimes even outperforms the PI-adaptive solver. Code is available at https://github.com/ellakemperman/adaptive-second-order-diffusion-solvers
We theoretically study the hallucination phenomena in two canonical diffusion samplers: the stochastic Denoising Diffusion Probabilistic Model (DDPM) and the deterministic Denoising Diffusion Implicit Model (DDIM). We analyze the reverse ODE (DDIM) and SDE (DDPM) for a Gaussian mixture target, proving that after a critical time τ, (a) DDIM can become stuck on the segment connecting the two nearest modes and (b) DDPM stochasticity helps it become unstuck from this region, thus avoiding hallucination. Our empirical validation verifies that DDPM has a significantly lower hallucination rate than DDIM when this region is entered. Building on our observations, we exhibit how using additional stochastic steps can help DDIM avoid hallucinations and offer new insights on how to design improved samplers.
Muhammad H. Ashiq, Samanyu Arora, Abhinav N. Harish +3
University of Wisconsin-Madison, Madison, WI, USA.
Diffusion-based generative models have achieved remarkable performance across various domains, yet their practical deployment is often limited by high sampling costs. While prior work focuses on training objectives or individual solvers, the broader sampling design problem, specifically solver selection and scheduling, remains largely governed by static heuristics. We propose SDM, a principled, training-free sampling framework that adapts both the numerical solver and the timestep schedule to the intrinsic properties of the diffusion trajectory. By analyzing the PF-ODE dynamics, we show that velocity variation is small in high-noise stages and increases near the data manifold, identifying intervals where solver order is most consequential. In parallel, we introduce an offline-calibrated adaptive scheduling method that explicitly controls the local Wasserstein discretization error and projects the calibrated trajectory to a prescribed NFE budget. We further extend the formulation to a mixed-transition Wasserstein error bound, providing a unified error-propagation view of adaptive scheduling and solver selection within the overall SDM framework. Across standard benchmarks, with extensions to modern ODE samplers, high-resolution synthesis, and text-to-image generation, SDM achieves improved sample quality compared to baseline methods, attaining an FID of 1.93 on CIFAR-10, 2.41 on FFHQ, and 1.98 on AFHQv2, with a reduced number of function evaluations compared to existing samplers. Our code is available at https://github.com/aiimaginglab/sdm.
Sangwoo Jo, Sungjoon Choi
Department of Artificial Intelligence, Korea University, Seoul, South Korea.