Speed-accuracy relations for diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport

Kotaro Ikeda; Tomoya Uda; Daisuke Okanohara; Sosuke Ito

cond-mat.stat-mechJul 5, 2024

Speed-accuracy relations for diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport

Authors: Kotaro Ikeda, Tomoya Uda, Daisuke Okanohara, Sosuke Ito

Organizations: Department of Mathematical Engineering and Information Physics, School of Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan · Department of Earth and Planetary Physics, School of Science, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan · Preferred Networks Inc., 1-6-1, Otemachi, Chiyoda-ku, Tokyo 100-0004, Japan · Universal Biology Institute, Graduate School of Science, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan

Abstract

We discuss a connection between a generative model, called the diffusion model, and nonequilibrium thermodynamics for the Fokker-Planck equation, called stochastic thermodynamics. Using techniques from stochastic thermodynamics, we derive the speed-accuracy relations for diffusion models, which are inequalities that relate the accuracy of data generation to the entropy production rate. This relation can be interpreted as the speed of the diffusion dynamics in the absence of the non-conservative force. From a stochastic thermodynamic perspective, our results provide quantitative insight into how best to generate data in diffusion models. The optimal learning protocol is introduced by the geodesic of space of the 2-Wasserstein distance in optimal transport theory. We numerically illustrate the validity of the speed-accuracy relations for diffusion models with different noise schedules and different data. We numerically discuss our results for optimal and suboptimal learning protocols. We also demonstrate the applicability of our results to data generation from the real-world image datasets.

Figures & tables

Figure 1: The organization of this paper. In Sec. II , we explain the basic concepts of diffusion models and discuss their methods, including score-based generative modeling and flow-based generative modeling. In Sec. III , we discuss stochastic thermodynamics and optimal transport theory, and introduce the entropy production rate. We also introduce several thermodynamic trade-off relations, such as thermodynamic uncertainty relations and speed limits, which provide the minimum entropy production in a finite time via optimal transport. In Sec. IV , We derive our main results in analogy to thermodynamic trade-off relations. Our main results reveal the essential role of the entropy production rate in diffusion models. Noise schedules based on optimal transport will lead to the most robust generation in diffusion models, and existing noise schedules such as the conditional optimal transport schedule and the cosine schedule are regarded as suboptimal noise schedules.

Figure 2: Illustration of diffusion models in the case of image data. The forward process, denoted by $\mathcal{P}_{t}(\bm{x})$ , is a stochastic process which adds noise to the training data sampled from $q(\bm{x})$ . By inverting this forward process in time, we prepare the reverse process $\mathcal{P}^{\dagger}_{\tilde{t}}(\bm{x})$ . The generated data sampled from $p(\bm{x})$ is constructed by an estimated reverse process $\tilde{\mathcal{P}}^{\dagger}_{\tilde{t}}(\bm{x})$ that mimics the reverse process. The image in the figure is taken from the dataset [ 91 ] .

Figure 3: Illustration of the speed-accuracy relation for diffusion models [Eq. ( 79 )] in the absence of the non-conservative force

Figure 4: Summary of the main results. We show a hierarchy between the inequalities, and a generalization for incomplete estimation discussed in Sec. V.1 .

Figure 5: Examples of the forward process and the estimated reverse process according to three noise schedules: the cosine schedule [Eq. ( 32 )], the cond-OT schedule [Eq. ( 28 )], and the optimal transport which gives the dynamics along the geodesic in the space of the 2-Wasserstein distance. These figures show the time evolution of samples from a $2$ -dimensional Swiss-roll dataset ( $n_{\rm{d}}=2$ ) in the flow-based generative modeling. In all noise schedules, we used the same sample for the initial distribution of the estimated reverse process $\tilde{\mathcal{P}}_{\tau}(\bm{x})$ . Compared with the cosine schedule and the cond-OT schedule, the noise in the generated figure is relatively small when we consider the optimal transport. This result is consistent with the optimality based on the speed-accuracy relations for diffusion models.

Figure 6: Numerical calculations of the speed-accuracy relations for diffusion models [Eqs. ( 79 ), ( 80 ) and ( 83 )]. (a) The time evolution of the probability density functions in the forward process and the estimated reverse process with the cosine schedule, the cond-OT schedule and the optimal transport (OT). Here $t$ denotes time in the forward direction, and we show the probability density functions at $t=0$ , $t=\tau/3$ , $t=2\tau/3$ and $t=\tau$ . The final state of the forward process is $\mathcal{P}_{\tau}(\bm{x})=\mathcal{N}(\bm{x}|0,1)$ . We used a Gaussian distribution with a different mean $\tilde{\mathcal{P}}_{0}^{\dagger}(\bm{x})=\mathcal{N}(\bm{x}|1,1)$ as the initial condition of the estimated reverse process. (b-e) We show the situation where the initial condition of the estimated reverse process $\tilde{\mathcal{P}}_{0}^{\dagger}(\bm{x})$ is (b) a Gaussian distribution with a different mean, (c) a Gaussian distribution with a different variance, (d) a Gaussian mixture distribution, (e) a uniform distribution. The plot at the top left of each section shows the comparison between $\tilde{\mathcal{P}}_{0}^{\dagger}(\bm{x})$ and $\mathcal{P}_{\tau}(\bm{x})=\mathcal{N}(\bm{x}|0,1)$ , where $\mathcal{N}(\bm{x}|0,1)$ is shown in gray. In the top right-hand section, we show the bar graph which displays the quantities $\eta=(\Delta\mathcal{W}_{1})^{2}/(\tau D_{0})$ , $c_{\rm l}=\int_{0}^{\tau}dt\;[v_{\rm loss}(t)]^{2}$ , $c_{2}=\int_{0}^{\tau}dt\;[v_{2}(t)]^{2}$ , and this figure shows the validity of the inequality $\eta\leq c_{1}\leq c_{2}$ [Eq. ( 83 )]. In the bottom row we show the time evolution of $[v_{2}(t)]^{2}$ and $[v_{\rm loss}(t)]^{2}$ , and this figure shows the validity of the inequality $[v_{2}(t)]^{2}\geq[v_{\rm loss}(t)]^{2}$ [Eq. ( 80 )]. Since $[v_{2}(t)]^{2}$ and $c_{2}$ depend only on the forward process, these values are consistent within different initial conditions in (b)-(e). See also Table 3 in Appendix G for more detailed explanations, including the probability density function $\tilde{\mathcal{P}}_{0}^{\dagger}(\bm{x})$ used in (b)-(e).

Initial conditions	Noise schedules	Values of $\eta$
Gaussian distribution with a different mean	Cosine Cond-OT OT	$9.188\times 10^{-2}$ $8.781\times 10^{-2}$ $8.537\times 10^{-2}$
Gaussian distribution with a different variance	Cosine Cond-OT OT	$1.078\times 10^{-1}$ $1.047\times 10^{-1}$ $1.022\times 10^{-1}$
Gaussian mixture distribution	Cosine Cond-OT OT	$2.764\times 10^{-2}$ $2.674\times 10^{-2}$ $2.611\times 10^{-2}$
Uniform distribution	Cosine Cond-OT OT	$2.142\times 10^{-2}$ $2.068\times 10^{-2}$ $2.033\times 10^{-2}$

Table 1: The values of the response function

\eta=(\Delta\mathcal{W}_{1})^{2}/(\tau D_{0})

in the numerical calculations [in Fig. 6 ]. Since we computed

3\times 10^{8}

samples, we display the value with four significant digits.

Figure 7: Numerical experiments of the speed-accuracy relations for diffusion models [Eqs. ( 77 ), ( 80 ) and ( 81 )] in two-dimensional Swiss-roll situation. (a)-(c) The time evolution of the probability density function in the forward process and the estimated reverse process with the noise schedules using $g=0,3,5$ in Eq. ( 99 ). The value $g$ stands for the degree of non-conservativity and $g=0$ denotes a situation with conservative external force $\bm{f}_{t}(\bm{x})$ . (d) The time evolution of the probability density function with optimal transport noise schedule. In (a)-(d) $t$ denotes time in the forward direction, and we show the probability density functions at $t=0$ , $t=\tau/3$ , $t=2\tau/3$ and $t=\tau$ . Each axis of the color map denotes the axis of the data space $\mathbb{R}^{n_{\rm d}}$ . Throughout this experiment, we used a Gaussian distribution with different variance $\tilde{\mathcal{P}}_{0}^{\dagger}(\bm{x})=\mathcal{N}(\bm{0},(0.9)^{2}\mathsf{I}$ ) as the initial condition of the estimated reverse process. (e) Temporal evolution of $T_{t}\dot{S}_{t}^{\rm tot}$ and $[v_{\rm loss}(t)]^{2}$ proving the validity of $[v_{\rm loss}(t)]^{2}\leq T_{t}\dot{S}_{t}^{\rm tot}$ [Eq. ( 80 )]. (f) A bar graph showing the quantities $\eta=(\Delta\mathcal{W}_{1})^{2}/(\tau D_{0})$ , $c_{\rm l}=\int_{0}^{\tau}dt\>[v_{\rm loss}(t)]^{2},w_{\rm D}=\int_{0}^{\tau}dt\>T_{t}\dot{S}_{t}^{\rm tot}$ . This figure shows the relationship $\eta\leq c_{\rm l}\leq w_{\rm D}$ [Eq. ( 81 )]. For detailed explanations see Appendix G .

Initial conditions	Noise schedules	Values of $\eta$
Gaussian distribution with a different variance	$g=0$ $g=3$ $g=5$ OT	$9.301\times 10^{-2}$ $3.430\times 10^{-1}$ $5.615\times 10^{-1}$ $7.221\times 10^{-6}$

Table 2: The values of the response function

\eta=(\Delta\mathcal{W}_{1})^{2}/(\tau D_{0})

in the numerical calculations [Fig. 7 ]. Since we computed

1\times 10^{8}

samples, we display the value with four significant digits.

Figure 8: The application of diffusion models using the real-world image datasets and the validity of Eq. ( 79 ) on the latent space. (a) The forward process and the estimated reverse process for the downscaled version of the CelebA HQ dataset [ 119 ] . (b) The forward process and the estimated reverse process for the bedrooms subset of the LSUN dataset [ 120 ] . (c) The estimated values of $\eta=(\Delta\mathcal{W}_{1})^{2}/(\tau D_{0}),\;c_{2}=\int_{0}^{\tau}dt[v_{2}(t)]^{2}$ computed on the latent space. The error bars shows the standard deviation among 5 calculations using $3\times 10^{4}$ samples for each calculations. $n_{\rm d}=4096$ denotes the dimension of the latent space. The inequality $c_{2}\geq\eta$ [Eq. ( 79 )] on the latent space is still valid for realistic applications of diffusion models using the real-world image datasets.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Initial conditions	Probability density functions
Gaussian distribution with a different mean	$q(\bm{x})=\mathcal{N}(\bm{x}\|1.0,1.0)$
Gaussian distribution with a different variance	$q(\bm{x})=\mathcal{N}(\bm{x}\|0.0,0.8^{2})$
Gaussian mixture distribution	$q(\bm{x})=\frac{1}{3}[\mathcal{N}(\bm{x}\|\frac{-1}{0.94},\left(\frac{0.3}{0.94}\right)^{2})+\mathcal{N}(\bm{x}\|0,\left(\frac{0.8}{0.94}\right)^{2})+\mathcal{N}(\bm{x}\|\frac{1}{0.94},\left(\frac{0.3}{0.94}\right)^{2})$
Uniform distribution	$q(\bm{x})=\frac{1}{2\sqrt{3}}\;(\bm{x}\in[-\sqrt{3},\sqrt{3}])$

Appendix

Table 3: The initial conditions used in the numerical calculations [Fig. 6 ]. Here

\mathcal{N}(\bm{x}|\mu,\sigma^{2})

denotes the

1

-dimensional Gaussian distribution with mean

\mu

and variance

\sigma^{2}

Figure 9: The time evolution of the probability density functions in the forward and estimated reverse processes, for each condition used in the numerical calculations [Fig. 6 ].

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,192 papers

Some remarks on the connection between generative diffusion and optimal transport
Oct 9, 2026Giulio Biroli, Marc MézardDiffusion Models Generative Modeling
Laboratoire de Physique de l’Ecole Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris-Diderot, Sorbonne Paris Cité, Paris, France · Department of Computing Sciences, Bocconi University, Milano, Italy
Noise Schedule Design for Diffusion Models: An Optimal Control Perspective
May 21, 2026Seo Taek Kong, Weina Wang, R. SrikantDiffusion Sampling Diffusion-Based Image Generation
ECE & CSL, University of Illinois Urbana-Champaign · Computer Science Department, Carnegie Mellon University · ECE, CSL & NCSA, University of Illinois Urbana-Champaign
Stochastic Thermodynamics and SDE-based Generative Models
Jun 13, 2026Yaowen ZhangSchrödinger Bridges Diffusion Models
Department of Physics, The Hong Kong University of Science and Technology

Oct 9, 2026J/K move · Enter open · S save

Some remarks on the connection between generative diffusion and optimal transport

Giulio Biroli, Marc Mézard

Laboratoire de Physique de l’Ecole Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris-Diderot, Sorbonne Paris Cité, Paris, France · Department of Computing Sciences, Bocconi University, Milano, Italy

Diffusion Models Generative Modeling Optimal Transport

Open paper AskPDF arXiv

We discuss three simple connections between generative diffusion and optimal transport. First, the joint law of the data and of their Ornstein-Uhlenbeck noising is, at every time, the solution of an entropic optimal-transport problem, at a temperature set by the noise level. Backward diffusion can consequently be read as an annealing which gradually decreases the entropic regularization. Second, we study the speciation transition--how the modes of the data distribution emerge during generation--and compare how it happens in diffusion and flow matching with respect to optimal transport. Using Gaussian mixtures as high-dimensional example, we show that the dynamical instability through which diffusion and flow matching separate the modes becomes, for optimal transport, an exponentially thin boundary layer at the beginning of generation, which is rounded by entropic regularization. Third, considering a generic forward dynamics with a general confining potential, we show that the backward process driven by the score also satisfies a drift-based variational principle whose minimal cost is the entropy difference between initial and final distributions.

Abstract

Diffusion Models

Explore similar work

Some remarks on the connection between generative diffusion and optimal transport

Noise Schedule Design for Diffusion Models: An Optimal Control Perspective

Stochastic Thermodynamics and SDE-based Generative Models