Organizations: D´epartement d’informatique, ENS, Universit´e PSL, Paris, France · Capital Fund Management, Paris, France · Coll`ege de France, Paris, France · Flatiron Institute, New York, USA · Courant Institute of Mathematical Sciences, New York University, New York, USA · ML Lab, Capital Fund Management, Paris, France
Generating samples from limited information is a fundamental problem across scientific domains. Classical maximum entropy methods provide principled uncertainty quantification from moment constraints but require sampling via MCMC or Langevin dynamics, which typically exhibit exponential slowdown in high dimensions. In contrast, generative models based on diffusion and flow matching efficiently transport noise to data but offer limited theoretical guarantees and can overfit when data is scarce. We introduce Moment Guided Diffusion (MGD), which combines elements of both approaches. Building on the stochastic interpolant framework, MGD samples maximum entropy distributions by solving a stochastic differential equation that guides moments toward prescribed values in finite time, thereby avoiding slow mixing in equilibrium-based methods. We formally obtain, in the large-volatility limit, convergence of MGD to the maximum entropy distribution and derive a tractable estimator of the resulting entropy computed directly from the dynamics. Applications to financial time series, turbulent flows, and cosmological fields using wavelet scattering moments yield estimates of negentropy for high-dimensional multiscale processes.
Figures & tables
Approach
Input
Max-ent guarantee
Moment control
Sampling
Maximum entropy (classical)
Moments m
✓
✓
Equilibrium (MCMC)
Diffusion models
Dataset (xi)
×
×
Non-equilibrium
Moment Guided Diffusion
Dataset (xi)
✓
✓
Non-equilibrium (guided)
Table 1: Comparison of sampling approaches for complex distributions.
Figure 1: Illustration of trajectories ( in blue or red) of Xt satisfying Equation ( 9 ) for an interpolant It defined with αt=πt/2 between white noise Z and a bimodal unbalanced Gaussian mixture X , for σ=1 . We display in gray in the background the density of It . When t goes to 0 , the modes progressively disappear. At early times t , particles evolve freely in space, but they become trapped in the modes when the density pt becomes bimodal. Red particles are confined in the upper mode and blues in the lower one.
Figure 2: Convergence of MGD towards the maximum entropy bimodal distribution p∗(x)=Z−1e−54(x4−5x2−x/2) for X∼p=p∗ . Left column: moment function ϕ(x)=(x,x2,x3,x4) . Right column: ϕ(x)=(x2,logp(x)) . (a,d) Log-density logp∗(x) (dashed) and logp1σ(x) for increasing σ (blue to red). (b,e) Maximum entropy H(p∗) (red line), sampled entropy H(p1σ) (blue dots), and lower bound H∗σ from ( 37 ) (black dots) versus σ2 . (c,f) Entropy gaps H(p∗)−H(p1σ) (blue) and H(p∗)−H∗σ (black) versus σ2 ; the dashed line shows σ−2 decay.
Figure 3: Convergence of MGD towards the Laplacian maximum entropy distribution p∗(x)=21e−∣x∣ for X∼p=p∗ . (a) Log-density logp∗(x) (dashed) and logp1σ(x) for increasing σ (blue to red). (b, top) Maximum entropy H(p∗) (red line), sampled entropy H(p1σ) (blue dots), and lower bound H∗σ from ( 37 ) (black dots) versus σ2 . (b, bottom) Entropy gaps H(p∗)−H(p1σ) (blue) and H(p∗)−H∗σ (black) versus σ2 ; the dashed line shows σ−2 decay.
Figure 4: (a) Log-density logp∗(x) for p∗(x)=Zβ−1e−β(x4−5x2−x/2) with increasing β (blue to red). The two modes are separated by a barrier of height proportional to β . (b) Number of discretization steps nsteps required to reach a fixed Kullback–Leibler divergence from p∗ , for MALA (red) and MGD (green), as a function of β . For MALA, nsteps grows exponentially with β ; for MGD, it remains nearly constant.
Figure 5: (a): One-dimensional Morlet wavelet ψ . The wavelet is a complex function whose real and imaginary parts are respectively in blue and red. (b): real (left) and imaginary (right) parts of a two-dimensional Morlet wavelet.
Figure 6: (a) S&P 500 daily log-returns ( d=6060 ) from January 2000 to February 2024. (b) Sample generated by MGD with σ2=5.5 , using r=217 empirical wavelet scattering moments ( 49 ) computed from (a). Intermittency is reproduced.
Figure 7: (a) Negentropy estimate ΔH∗σ from ( 43 ) versus σ2 for S&P 500 log-returns. (b) Convergence of H∗σ , measured by d−1(H∗σmax−H∗σ) with σmax2=5.5 , versus σ2 . The plain line shows σ−2 decay.
Figure 8: Histograms of rolling volatility vol(u) computed over w=5 days. Dashed: S&P log-returns X . Red to green: MGD samples X1σ for increasing σ2 . Black: Gaussian process with the same covariance. At σ2=5.5 , the histogram of X1σ matches the S&P and differs from the Gaussian.
Figure 9: (a) Cosmic web field: 2D slice from a 3D dark matter simulation. (b) Turbulent vorticity field from 2D incompressible Navier–Stokes. Both images are 128×128 pixels. (c,d) MGD samples with wavelet scattering moments at σ2=5.5 .
Figure 10: (a) Negentropy estimate ΔH∗σ from ( 43 ) versus σ2 for cosmic web (blue) and turbulence (red). (b) Convergence of H∗σ , measured by d−1(H∗σmax−H∗σ) with σmax2=5.5 , versus σ2 . Plain lines show σ−2 decay.
Dataset
Estimated Normalized Negentropy
Laplacian
0.07
S&P 500
0.05
Cosmic Web
0.07
2D Turbulence
0.34
Table 2: Negentropy estimate ΔH∗σ at σ=σmax .
Figure 11: Histograms of finest-scale wavelet coefficients Re(X1λ) for λ=ξ ( j=0 , ℓ=0 ) of cosmic web samples from the scattering MGD model, with σ2∈{0,0.1,5.5} . Dashed black: original data. All histograms are computed over 500 samples. Larger σ2 yields better tail reproduction; small σ2 produce more regular samples which have too many small wavelet coefficients, as in Figure 2 (a).
Figure 12: MGD with large model error on CelebA faces ( 64×64 ). (a) Original sample. (b) MGD sample with wavelet scattering moments at σ2=1 . (c) Negentropy estimate ΔH∗σ from ( 43 ) versus σ2 . (d) Convergence of H∗σ , measured by d−1(H∗σmax−H∗σ) with σmax2=8.5 ; the line shows σ−2 decay.
Diffusion models generate samples by denoising along the score of a perturbed target distribution. In practice, one trains a neural diffusion model, which is computationally expensive. Recent work suggests that score matching implicitly smooths the empirical score, and that this smoothing bias promotes generalization by capturing low-dimensional data geometry. We propose moment-matched score-smoothed overdamped Langevin dynamics (MM-SOLD), a training-free interacting particle sampler that enforces the target moments throughout the sampling trajectory. We prove that, in the large-particle limit, the empirical particle density converges to a deterministic limit whose one-particle stationary marginal is a Gibbs--Boltzmann density obtained by exponentially tilting a naive score-smoothed diffusion target. The mean and covariance of this distribution agree with the empirical moments of the training data. Experiments on 2D distributions and latent-space image generation show that MM-SOLD enables fast, robust, training-free sampling on CPUs, with sample fidelity and diversity competitive with neural diffusion baselines.
Zhenyu Yao, Daniel Paulin
College of Computing and Data Science Nanyang Technological University
Diffusion models perform remarkably well on high-dimensional data such as images, often using only a modest number of reverse-time steps. Despite this practical success, existing convergence theory does not fully explain why such samplers remain efficient in high dimensions. Many prior KL guarantees bound the discretization error in terms of the ambient dimension, while other improved results replace this dependence using intrinsic-dimensional or geometric structure assumptions. In this work, we develop an alternative information-theoretic perspective on diffusion sampler convergence. We prove that, for Gaussian mixture targets, the discretization error is controlled by the Shannon entropy of the latent mixture component rather than by the ambient dimension. Consequently, the leading step complexity scales linearly with latent entropy and depends only logarithmically on the second moment of the data. Our analysis also extends to discrete target distributions, where the relevant complexity is the entropy of the target rather than the dimension of the embedding space. These results suggest that diffusion sampling can remain efficient in high-dimensional spaces when the data distribution admits a compact latent representation, as is widely believed to be the case for natural images.
Ahmad Aghapour, Erhan Bayraktar
Department of Mathematics, University of Michigan, Ann Arbor, 48109
Guided diffusion sampling relies on approximating often intractable likelihood scores, which introduces significant noise into the sampling dynamics. We propose using adaptive moment estimation to stabilize these noisy likelihood scores during sampling. Despite its simplicity, our approach achieves state-of-the-art results on image restoration and class-conditional generation tasks, outperforming more complicated methods, which are often computationally more expensive. We provide empirical analysis of our method on both synthetic and real data, demonstrating that mitigating gradient noise through adaptive moments offers an effective way to improve alignment.
Christian Belardi, Justin Lovelace, Kilian Q. Weinberger +1