Adaptive Second-Order Solvers for Fast Stochastic Diffusion Sampling

Ella Kemperman; Luca Ambrogioni

cs.LGOct 2, 2026

Adaptive Second-Order Solvers for Fast Stochastic Diffusion Sampling

Authors: Ella Kemperman, Luca Ambrogioni

Organizations: Radboud University

Abstract

Diffusion models rely on numerical solvers requiring time-discretization, which has a large influence on the tradeoff between sampling cost and quality. However, the computational difficulty of the reverse process varies along the sampling trajectory and across data distributions, making the choice of discretization important. We adapt proportional-integral (PI) step-size control to diffusion, using our diffusion noise-normalised error estimator. Unlike existing adaptive methods in diffusion that respond only to the current error, the PI solver also incorporates the previous error, yielding smoother step adaptation. We further show that these per-sample trajectories exhibit shared structure and can be aggregated into a fixed schedule that retains much of the benefit of adaptive sampling. We evaluate both approaches on natural-image and language datasets, in terms of quality, measured by FID at a matched number of neural network evaluations (NFE), comparing them with widely used stochastic solvers and schedules. For images, our fixed discretization outperforms the commonly used EDM schedule in terms of sample quality when used with the stochastic Heun sampler, and with the EDM-churn sampler at low NFE. Additionally, our PI adaptive solver obtains better FID than most stochastic and adaptive baselines, although it does not beat the EDM-churn sampler at low NFE. Moreover, we find our solver outperforms both the EDM and the entropy schedule on language diffusion at low-to-medium NFE in terms of perplexity, with the drawback of lower token entropy. Lastly, we find that the benefit of per-sample adaptivity is problem-dependent. It is highly beneficial in 1D toy examples, while only marginal for image and language data, where the average schedule sometimes even outperforms the PI-adaptive solver. Code is available at https://github.com/ellakemperman/adaptive-second-order-diffusion-solvers

Figures & tables

Panel of Figure 1 — Figure 1: The difficulty of generative diffusion differs across time, requiring different step-sizes at different time-points, which our PI solver determines (a). These per-sample paths share structure and can then be aggregated into an average path usable for sampling (b).

Panel of Figure 2 — Figure 2: The Wasserstein distance against the NFE for different SDE solvers applied to diffusion on a mixture of Gaussians (a). The cumulative computational effort allocated ( $\frac{i}{N}$ ) plotted against diffusion time $t$ at $\tau_{abs}\approx 0.02656$ applied to diffusion on a Gaussian mixture (b).

Solver	Schedule	FFHQ			ImageNet 64x64
Solver	Schedule	49 NFE	75 NFE	99 NFE	49 NFE	75 NFE	99 NFE
Euler-Marayuma	EDM	$318.96\!\pm\!0.07$	$171.55\!\pm\!0.27$	$91.83\!\pm\!0.17$	$187.66\!\pm\!0.23$	$101.48\!\pm\!0.40$	$58.23\!\pm\!0.15$
Euler-Marayuma	Entropy	$226.70\!\pm\!0.20$	$81.33\!\pm\!0.20$	$43.98\!\pm\!0.20$	$90.54\!\pm\!0.31$	$36.55\!\pm\!0.06$	$18.65\!\pm\!0.03$
Stochastic Heun	EDM	$274.33\!\pm\!0.23$	$23.55\!\pm\!0.14$	$3.58\!\pm\!0.05$	$167.75\!\pm\!0.23$	$20.98\!\pm\!0.26$	$2.71\!\pm\!0.02$
	Entropy	$348.81\!\pm\!0.15$	$3.26\!\pm\!0.02$	$3.54\!\pm\!0.05$	N/A	N/A	N/A
	PI Static (ours)	$137.69\!\pm\!0.21$	$3.26\!\pm\!0.06$	$3.22\!\pm\!0.07$	$66.93\!\pm\!1.10$	$\mathbf{1.48}\!\pm\!0.02$	$\mathbf{1.59}\!\pm\!0.02$

Table 1: The mean FID (

\downarrow

) and standard deviation of a number of solver-schedule combinations at different NFE for ImageNet-64 and FFHQ.

Panel of Figure 3 — Figure 3: The cumulative effort allocation ( $\frac{i}{N}$ ), plotted against diffusion noise $\sigma$ , arrived at by the PI solver at 75 NFE for ImageNet-64 and FFHQ, and 127 NFE for LM1B, compared to the EDM schedule and LM1B entropic schedule (a). Comparison of ImageNet-64 samples generated using different solver-schedule combinations at 49, 75, and 99 NFE, with the same random seed (b).

Panel of Figure 4 — Figure 4: The NFE (a) and the image quality measured by MIND (b) as a function of $\tau_{rel}$ and $\tau_{abs}$ measured by generating 64 ImageNet-64 images at each combination of $\tau_{rel},\tau_{abs}$ .

Sampler	Schedule	63 NFE		127 NFE		255 NFE
Sampler	Schedule	Perplexity	Entropy	Perplexity	Entropy	Perplexity	Entropy
DDIM	Entropic	$128.26\!\pm\!1.62$	$4.35\!\pm\!0.00$	$87.03\!\pm\!0.28$	$4.34\!\pm\!0.01$	$\mathbf{60.02}\!\pm\!0.83$	$4.31\!\pm\!0.00$
	EDM	$233.26\!\pm\!3.13$	$\mathbf{4.36}\!\pm\!0.01$	$159.14\!\pm\!3.36$	$\mathbf{4.35}\!\pm\!0.00$	$109.00\!\pm\!0.31$	$\mathbf{4.34}\!\pm\!0.01$
	PI Static (ours)	$165.56\!\pm\!0.64$	$4.35\!\pm\!0.00$	$121.67\!\pm\!0.68$	$4.34\!\pm\!0.01$	$83.46\!\pm\!0.77$	$4.33\!\pm\!0.01$
EDM-churn	Entropic	$152.85\!\pm\!1.37$	$4.33\!\pm\!0.00$	$115.95\!\pm\!0.97$	$4.33\!\pm\!0.00$	$84.84\!\pm\!0.66$	$4.31\!\pm\!0.01$
	EDM	$214.76\!\pm\!0.29$	$4.33\!\pm\!0.01$	$156.30\!\pm\!2.05$	$4.33\!\pm\!0.01$	$116.49\!\pm\!1.59$	$4.32\!\pm\!0.01$

Table 2: The perplexity (

\downarrow

) and unigram entropy (

\uparrow

) of solver-schedule combination at different NFE for LM1B.

Configuration	Components	49 NFE	75 NFE	99 NFE
Config A (base)	PI Control + Mixed Noise Tolerance	84.09	3.36	3.11
Config B	I Control + Mixed Noise Tolerance	125.81	4.04	3.53
Config C	PI Control + Mixed Absolute Tolerance	149.23	66.7	49.77
Config D	PI Control + Non-Mixed Tolerance	357.64	319.14	298.59

Table 3: PI solver ablations on FFHQ, measuring FID (

\downarrow

) of the PI solver at different NFE with varying components changed or removed.

Table 4: A PI schedule generated by the PI solver on FFHQ (images) compared to one generated on LM1B (language), evaluated in terms of FID (

\downarrow

) at different NFE on FFHQ (a) and in terms of generative perplexity at different NFE on LM1B (b).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: The 1-dimensional sensitivity of the $\alpha$ (a), $\tau_{rel}$ (b), $k_{i}$ (c), and $k_{p}$ (d) hyperparameters of the PI solver with regards to the minimum Wasserstein distance of the Wasserstein distance - NFE tradeoff, evaluated on a 1D multimodal Gaussian with varying means, variances, and weights.

Figure 6: Influence of varying $h_{0}$ on the NFE - Wasserstein distance trade-off of the PI solver on a 1D multimodal Gaussian with varying means, variances, and weights.

Panel of Figure 7 — Figure 7: The results of the initial grid-search over the absolute tolerance $\tau_{abs}$ and relative tolerance $\tau_{rel}$ of the PI solver, measuring NFE and the MIND error metric.

	1	2	3	4	5	6	7	8	9	10
$\pi$	1	3	1	3	1	2	1	3	2	1
$\mu$	-12	-10	-8	-5	0	4	9	13	15	17
$\sigma$	0.5	1.5	0.2	2	0.3	1.5	0.1	1	0.2	0.1

Appendix

Table 5: Parametrization of the varying Gaussian.

Parameter	Value
$\tau_{r}$	0
$\alpha$	0.9
$\alpha_{min}$	0.2
$\alpha_{max}$	5
$h_{0}$	0.15
$t_{ode}$	0

Appendix

Table 6: Hyperparameters used for the PI solver for 1D Gaussians using the VP SDE.

Parameter	PI solver			Gotta Go Fast
Parameter	49 NFE	75 NFE	99 NFE	49 NFE	75 NFE	99 NFE
$\tau_{abs}$	0.005	0.005	0.005	0.0078	0.0078	0.0078
$\tau_{rel,\text{ImageNet-64}}$	1.09	0.87	0.72	25.5	17	13.7
$\tau_{rel,\text{FFHQ}}$	1.15	0.92	0.72	25.4	17.15	13.7
$\alpha$	0.9	0.9	0.9	0.7	0.7	0.7
$\alpha_{min}$	0.2	0.2	0.2	N/A	N/A	N/A

Appendix

Table 7: Hyperparameters used for image generation of the PI solver, and the adaptive solver devised by Jolicoeur-Martineau et al. [7] (Gotta Go Fast)

Parameter	PI Solver
Parameter	63 NFE	127 NFE	255 NFE
$\tau_{a}$	0.005	0.005	0.005
$\tau_{r}$	0.0888	0.1017	0.0896
$\alpha$	0.9	0.9	0.9
$\alpha_{min}$	0.2	0.2	0.2
$\alpha_{max}$	5	5	5

Appendix

Table 8: Hyperparameters for text generation using the PI solver.

Configuration	Variable Name	49 NFE	75 NFE	99 NFE
Config C	$\tau_{rel}$	91	89	67
Config D	$\tau_{abs}$	8.5	5.3	4.5

Appendix

Table 9: PI solver ablations hyperparameters.

Panel of Figure 8 — Figure 8: ImageNet-64 Images generated using different SDE solvers using the EDM2-XXL FID optimized checkpoint at 49 function evaluations

Panel of Figure 9 — Figure 9: ImageNet-64 Images generated using different SDE solvers using the EDM2-XXL FID optimized checkpoint at 75 function evaluations

Panel of Figure 10 — Figure 10: ImageNet-64 Images generated using different SDE solvers using the EDM2-XXL FID optimized checkpoint at 99 function evaluations

Panel of Figure 11 — Figure 11: FFHQ Images generated using different SDE solvers using the EDM-FFHQ-64x64 unconditional VP checkpoint at 49 function evaluations

Panel of Figure 12 — Figure 12: FFHQ Images generated using different SDE solvers using the EDM-FFHQ-64x64 unconditional VP checkpoint at 75 function evaluations

Panel of Figure 13 — Figure 13: FFHQ Images generated using different SDE solvers using the EDM-FFHQ-64x64 unconditional VP checkpoint at 99 function evaluations

Explore similar work

Feb 13, 2026

Formalizing the Sampling Design Space of Diffusion-Based Generative Models via Adaptive Solvers and Wasserstein-Bounded Timesteps

Diffusion-based generative models have achieved remarkable performance across various domains, yet their practical deployment is often limited by high sampling costs. While prior work focuses on training objectives or individual solvers, the broader sampling design problem, specifically solver selection and scheduling, remains largely governed by static heuristics. We propose SDM, a principled, training-free sampling framework that adapts both the numerical solver and the timestep schedule to the intrinsic properties of the diffusion trajectory. By analyzing the PF-ODE dynamics, we show that velocity variation is small in high-noise stages and increases near the data manifold, identifying intervals where solver order is most consequential. In parallel, we introduce an offline-calibrated adaptive scheduling method that explicitly controls the local Wasserstein discretization error and projects the calibrated trajectory to a prescribed NFE budget. We further extend the formulation to a mixed-transition Wasserstein error bound, providing a unified error-propagation view of adaptive scheduling and solver selection within the overall SDM framework. Across standard benchmarks, with extensions to modern ODE samplers, high-resolution synthesis, and text-to-image generation, SDM achieves improved sample quality compared to baseline methods, attaining an FID of 1.93 on CIFAR-10, 2.41 on FFHQ, and 1.98 on AFHQv2, with a reduced number of function evaluations compared to existing samplers. Our code is available at https://github.com/aiimaginglab/sdm.

Sangwoo Jo, Sungjoon Choi

Department of Artificial Intelligence, Korea University, Seoul, South Korea.

Diffusion Model Sampling Diffusion Sampling Adaptive Sampling Diffusion Timestep Scheduling Diffusion Model Acceleration

Ask PDF

Feb 24, 2026

The Diffusion Duality, Chapter II: $Ψ$ -Samplers

Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on: https://s-sahoo.com/duo-ch2

Justin Deschenaux, Caglar Gulcehre, Subham Sekhar Sahoo

EPFL, Lausanne, Switzerland · Microsoft AI · Cornell Tech, NY

Diffusion Models Diffusion Sampling Diffusion Language Models

Ask PDF

Mar 4, 2026

Dual-Solver: A Generalized ODE Solver for Diffusion Models with Dual Prediction

Diffusion models achieve state-of-the-art image quality. However, sampling is costly at inference time because it requires a large number of function evaluations (NFEs). To reduce NFEs, classical ODE numerical methods have been adopted. Yet, the choice of prediction type and integration domain leads to different sampling behaviors. To address these issues, we introduce Dual-Solver, which generalizes multistep samplers through learnable parameters that continuously (i) interpolate among prediction types, (ii) select the integration domain, and (iii) adjust the residual terms. It retains the standard predictor-corrector structure while preserving second-order local accuracy. These parameters are learned via a classification-based objective using a frozen pretrained classifier (e.g., MobileNet or CLIP). For ImageNet class-conditional generation (DiT, GM-DiT) and text-to-image generation (SANA, PixArt-

α

), Dual-Solver improves FID and CLIP scores in the low-NFE regime (

3 \le

NFE

\le 9

) across backbones.

Soochul Park, Yeon Ju Lee

SteAI · MODULABS · Korea University

Diffusion Model Inference Acceleration Few-Step Diffusion Sampling

Ask PDF

Abstract

Explore similar work

Formalizing the Sampling Design Space of Diffusion-Based Generative Models via Adaptive Solvers and Wasserstein-Bounded Timesteps

The Diffusion Duality, Chapter II: ΨΨΨ-Samplers

Dual-Solver: A Generalized ODE Solver for Diffusion Models with Dual Prediction

The Diffusion Duality, Chapter II: $Ψ$ -Samplers