cs.LGAug 24, 2026

The Loss Floor of Denoising Score Matching: Fisher Geometry from Schrödinger Bridges

Authors: Avinash Raju, Kai Zhang

Organizations: Great Wall Motors Co. Ltd. Beijing, China · China Patent Information Center Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Denoising score matching trains diffusion models by regressing onto a conditional score, although generation ultimately requires the marginal score. The two objectives share the same population minimizer, but the conditional target remains random at fixed noisy state and introduces an irreducible excess in the training loss. We isolate this excess and show that, for a general corruption kernel under mild regularity assumptions, it is exactly the trace of the Fisher--Rao metric of the conditional endpoint family, integrated along the diffusion trajectory. This gives an exact conditional-variance decomposition of the denoising objective and identifies the information geometry observed in diffusion latent spaces as an intrinsic component of the training loss. We derive the result from a Schr"odinger bridge variational principle, in which the ideal objective arises as excess path-space relative entropy. For corruption diffusions, the Fisher term is proportional to the rate at which the noisy state loses mutual information about the clean data, separating the loss floor into an information flow determined by the data and a weight determined by the corruption schedule and objective. In the Gaussian case, this yields a closed form for the floor, recovers reparametrization invariance of the continuous-time objective, and relates its high-SNR divergence to the information dimension of the data. Finally, we show that raw losses obtained with different noise ranges or weightings need not rank models consistently because they contain different additive floors, and contrast the second-order geometry seen by training with the third-order conditional statistics entering numerical sampling error.

Figures & tables

Figure 1: Data/schedule separation of the loss floor. Left: the information spectrum S(λ)=λ MMSE(λ)S(\lambda)=\lambda\,\mathrm{MMSE}(\lambda) for two mixtures. Dotted lines mark the mode-separation scale λ=m−2\lambda=m^{-2} , dashed lines the within-mode scale λ=s−2\lambda=s^{-2} ; the plateau at large λ\lambda is the intrinsic dimension k=1k=1 . Right: the schedule density implied by Eq. ( 4.8 ), compared with a uniform-in- log⁡\log SNR allocation. Uniform allocation is correct only for featureless data.
Figure 2: The loss floor is an entropy. Left: the trace of the latent Fisher–Rao metric, equivalently the MMSE of endpoint estimation, along the flow. Right: the accumulated floor 12∫MMSE dλ\frac{1}{2}\int\mathrm{MMSE}\,d\lambda (solid) against the differential-entropy change Δh(xλ)\Delta h(\mathbf{x}_{\lambda}) (dashed), verifying Theorem 4.5 . Two-mode Gaussian mixture, entropies by quadrature.
Figure 3
Figure 5: Consequences of the decomposition. Left: raw training loss for two models of known quality on two SNR ranges; the ranking inverts. Centre: after subtracting the floor, the excess orders the models correctly on both ranges. Right: step placement for a first-order solver on the analytic mixture, mean endpoint error against the exactly integrated probability-flow ODE.
More in this topic

Score-Based Generative Modeling

Score-based generative modeling learns data-distribution gradients to generate samples through reverse diffusion, flows, and related stochastic dynamics.

See all 115 papers

Explore similar work

CardsList
Jun 4, 2026

Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors

Score-based diffusion models are typically trained by minimizing the L2L^2 score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the learned and target distributions. We show the L2L^2 score error is not the right intrinsic measure of marginal distributional quality: a learned diffusion model can incur arbitrarily large L2L^2 score error while perfectly matching the target distribution. By decomposing score errors into a gradient and a solenoidal component (a Helmholtz-Hodge decomposition), we identify the geometric reason behind this: only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible. We make this precise in three results. First, building on the corrected geometry, we prove an impossibility result: no monotone function of the L2L^2 score error can uniformly lower bound any divergence between the learned and target distributions. Second, we derive an upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error, tightening the standard Girsanov bound for generic score networks, and identifying its looseness as the cost of operating on path-space rather than marginal-space dynamics. Third, we give a tractable estimator of the gradient component via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L2L^2 error.

Naïl B. Khelifa, Richard E. Turner, Ramji Venkataramanan

University of Cambridge

Score MatchingHodge DecompositionScore-Based Generative ModelingDiffusion Models
AskPDF
Jul 5, 2026

Tightening the Score Matching Gap for Diffusion Models

Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation primarily rely on an Evidence Lower Bound (ELBO), which relates the Kullback-Leibler (KL) divergence of model samples to the score matching loss along the path, which serves as a tractable surrogate. The difference between sample quality and the score matching loss produced by this bound leads to the score matching gap, which is known to be tight in the worst-case but not descriptive of sample quality in general. In this work, we provide a theoretical analysis of this gap, developing tighter bounds for three metrics: KL divergence, reverse KL divergence, and Wasserstein distance, effectively exploiting the regularity of the class of score estimators. Our results suggest that the quality of the score approximation has more impact on closing the score matching gap for low noise scales. To obtain these bounds, our key technical insight is to exploit the contraction properties of the backward processes. In particular, we rely on entropy flows, logarithmic Sobolev inequalities and reflection couplings, rigorously linking the ergodicity of the Langevin diffusion to the score matching gap problem.

Benjamin Dupuis, Tyler Farghly, Maxime Haddouche +2

INRIA - CNRS - D´epartement d’Informatique de l’Ecole Normale Sup´erieure - PSL Research University, France · Ecole Polytechnique - CMAP, IP Paris, Palaiseau, France

Score MatchingScore-Based Generative ModelingDiffusion ModelsKullback-Leibler Divergence
AskPDF
Oct 26, 2026

Approximation Families and the Statistical Efficiency of Denoising Score Matching

We study score matching over finite-dimensional approximation spaces generated by gradients of eigenfunctions of the infinitesimal generator of a forward diffusion. We compare polynomial approximation under one-dimensional Ornstein-Uhlenbeck flow with trigonometric-polynomial approximation under heat flow on the torus. In the Ornstein-Uhlenbeck setting, we show that for every non-Gaussian initial distribution uniformly comparable to the standard Gaussian, the asymptotic score-estimation variance grows exponentially along a subsequence of polynomial degrees. Averaging the score-matching loss under the diffusion semigroup, which we call operator-informed score matching (OISM), reduces the variance at each fixed degree but does not eliminate this exponential instability at sufficiently small times. A local minimax lower bound shows that the same high-degree dependence arises when estimating the polynomial projection of the score, rather than being an artifact of OISM. In contrast, under heat flow on the torus, OISM over trigonometric polynomials admits finite-sample control of the score error along the forward diffusion and yields the classical minimax rate for estimating sufficiently smooth densities under expected Kullback-Leibler loss.

Brian Miyatake

Department of Statistics and Data Science, Yale University

Score MatchingScore-Based Generative ModelingDensity EstimationMinimax Estimation
AskPDF