Diffusion-based generative modeling has emerged as a general computational framework for simulating high-dimensional, structured data across various domains. Understanding its generative mechanisms is paramount for advancing model training and inference in a principled way. Here, we demonstrate that deterministic sampling trajectories share a universal geometric structure invariant to data modalities and diffusion model implementations. This structure can be largely explained by an exactly solvable model based on the optimal linear denoiser.
Figures & tables
Figure 1: Deterministic sampling trajectories share a universal structure across data modalities and diffusion model implementations. (a) Intermediate states along sampling trajectories, from random noise (left) to generated data (right). (b) Each trajectory projected onto its three leading principal components, with all starting points aligned (black point). (c) The trajectories aligned through an orthogonal transformation. Each axis in (b) and (c) is scaled independently to make the small deviation from a straight line visible.
Figure 2: The real trajectories, after an orthogonal transformation, closely follow the Gaussian hyperbolas for most eigenmode pairs (top row; legend: (i,j) and the eigenvalue gap Δλ=λi−λj ), except at the largest Δλ . The bottom row shows the distributions of the shape deviation, measured by the root-mean-square error (RMSE), between two independently initialized real trajectories and between a real trajectory and the Gaussian counterpart of another such trajectory. The table reports the deviation of each reference from the real trajectory before and after the transformation, normalized by the Gaussian chord deviation before the transformation. The Gaussian chord is the straight line from the initial noise to the endpoint of the Gaussian flow, the Brownian bridge is pinned to the same two endpoints, and the isotropic Gaussian flow replaces Σ with the scaled identity (trΣ/d)Id (see the Supplementary material for implementation details).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Distributions of the trajectory RMSE for independently initialized paths. The top row shows the error densities, and the bottom row shows the corresponding empirical cumulative distribution functions. In addition to the real–real and real–Gaussian comparisons in the bottom row of Figure 2 , we include the isotropic Gaussian flow, Gaussian chord, and Brownian bridge.
Figure 4: For a reference family M , we define Si(M)=meanj=id(Xi,Mj)/meanj=id(Xi,Xj), where d(⋅,⋅) is the optimally aligned trajectory RMSE and Mj uses the same initialization as real trajectory Xj . The curves show the distribution of this score over the 200 real trajectories. The dotted line marks Si=1 , matching the average real–real error; lower values mean closer agreement.
The performance of generative diffusion models is determined by the choice of the reference diffusion process connecting the empirical and prior distributions. Conventional approaches typically trade off simulation-free training against finite-time generation. We propose a framework for designing the reference process that achieves both simultaneously. The key idea is to prescribe tractable time-dependent conditional distributions and then construct the reference process realizing them as its marginals. This framework reveals that score matching is not fundamental to diffusion-model training but instead emerges naturally through reversal of the reference process. We further show that conditional flow matching arises as the small-noise limit of the proposed framework.
Kentaro Kaba, Masayuki Ohzeki, Yuki Sughiyama
Department of Physics, Institute of Science Tokyo, Meguro-ku, Tokyo 152-9551, Japan · 2Graduate School of Information Sciences, Tohoku University, Sendai, Miyagi 980-9564, Japan · 3Research and Education Institute for Semiconductors and Informatics, Kumamoto University, Kumamoto, Kumamoto 860-8555, Japan +1
Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.
Generative modelling involves two complementary tasks: learning a probabilistic approximation of an unknown data distribution from observations, and designing a tractable mechanism for sampling from this approximation. We present a stochastic-analysis perspective on score-based generative models, connecting ideas from classical sampling theory, score estimation, and diffusion time reversal. We first explain how score functions link Langevin sampling with score matching, and how Gaussian smoothing leads to denoising score matching and multiscale sampling methods. We then emphasize the distinction between dynamics preserving a fixed distribution and dynamics transporting a prescribed family of marginals. This motivates a path-space treatment of diffusion time reversal. Finally, we connect the ideal reverse diffusion with practical generative algorithms.
Stanislas Strasman
SU, LPSM · Laboratoire de Probabilités, Statistique et Modélisation, Sorbonne Université