Score-based generative models commonly employ linear forward diffusions with Gaussian reference distributions, limiting their ability to directly incorporate structural information into the noising dynamics. We introduce latent nonlinear denoising score matching (LNDSM), which combines nonlinear, structure-informed latent diffusions with the variational autoencoder (VAE) training framework. In our construction, the forward diffusion is designed to admit a prescribed reference distribution, such as a Gaussian mixture model, that reflects qualitative features of the latent data distribution. We derive a computable approximation to a cross-entropy upper bound in the VAE objective using Euler-Maruyama discretization and the resulting Gaussian conditional transitions. To stabilize training, we remove two zero-mean control variates whose variances become large for small time steps. The resulting objective enables joint training of the encoder, decoder, and latent score network. Numerical experiments on benchmark image datasets demonstrate the effectiveness of LNDSM in learning structured distributions, outperforming conventional latent score-based generative modeling in sample quality and mode balance across several settings, while achieving improved training efficiency. These results highlight the potential of nonlinear, structure-informed latent diffusions for generative modeling.
Figures & tables
Networks
# parameters (MNIST)
# parameters (CIFAR-10)
NVAE
207,648
299,518
NCSNpp
895,076
2,177,608
NVAE + NCSNpp
1,102,724
2,477,126
U-Net in NDSM-SGM
1,115,297
/
Table 1: Number of trainable parameters
Figure 1: The average log-likelihood plateaus at a terminal time of T=0.5 , indicating that the latent-space diffusion process has brought the latent variable distribution sufficiently close to the target GMM.
Figure 2: (a): For the MNIST dataset, LNDSM-SGM only needs a few training epochs to convergence, whereas the benchmark LSGM requires 500 epochs. The total training time of the proposed LNDSM-SGM is lower. (b): For the CIFAR-10 dataset, the total training time is capped at 48 hours. LNDSM-SGM is trained for 100 training epochs to convergence. The benchmark LSGM is trained for 436 epochs to convergence.
Figure 3: LSGM and LNDSM-SGM trained using the full MNIST dataset. (a) and (d): Snapshot and fraction of different digits of the training samples. (b) and (e): Snapshot and fraction of different digits of the 10,000 samples generated by the LSGM. The KL divergence from the fraction of the training samples to the fraction of the generated samples: 0.008. (c) and (f): Snapshot and fraction of different digits of the 10,000 samples generated by the LNDSM-SGM. The KL divergence of the fraction of the training samples from the fraction of the generated samples: 0.00114.
Table 2: FID and IS results on MNIST and CIFAR-10
Figure 4: LSGM and LNDSM-SGM trained using the CIFAR-10 dataset. (a) and (d): Snapshot and fraction of different classes of the training samples. (b) and (e): Snapshot and fraction of different classes of the 10,000 samples generated by the LSGM. The KL divergence from the fraction of the training samples to the fraction of the generated samples: 0.294. (c) and (f): Snapshot and fraction of different classes of the 10,000 samples generated by the LNDSM-SGM. The KL divergence from the fraction of the training samples to the fraction of the generated samples: 0.227.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: LSGM and LNDSM-SGM trained using the low data MNIST dataset. (a) and (d): Snapshot and fraction of different digits of the training samples. (b) and (e): Snapshot and fraction of different digits of the 10,000 samples generated by the LSGM. The KL divergence from the fraction of the training samples to the fraction of the generated samples: 0.00883. (c) and (f): Snapshot and fraction of different digits of the 10,000 samples generated by the LNDSM-SGM. The KL divergence from the fraction of the training samples to the fraction of the generated samples: 0.00439.
model
FID ( ↓ )
IS ( ↑ )
KL ( ↓ )
LSGM
27.1
9.12
0.00883
LNDSM-SGM
15.0
9.57
0.00439
Appendix
Table 3: Low data MNIST ( N=3000 )
Figure 6: LSGM and LNDSM-SGM trained using the full Approx.-C2-MNIST dataset. (a) and (d): Snapshot and fraction of different digits of the training samples. (b) and (e): Snapshot and fraction of different digits of the 10,000 samples generated by the LSGM. (c) and (f): Snapshot and fraction of different digits of the 10,000 samples generated by the LNDSM-SGM.
Score-based generative modeling learns data-distribution gradients to generate samples through reverse diffusion, flows, and related stochastic dynamics.
Diffusion models generate high-dimensional data with remarkable quality, yet how their training efficiently learns the score function, bypassing the curse of dimensionality when data is supported on low-dimensional manifolds, remains theoretically unexplained. We identify a collapse-and-refine mechanism driven by the geometry of the score function itself: at small noise scales, the diverging singularity of the score drives a rapid dimensional collapse of the induced denoising map onto the data manifold projection; at moderate noise scales, training refines the intrinsic density on the learned manifold. We instantiate this principle as Score-induced Latent Diffusion (SiLD), a two-stage framework in which both manifold learning and density estimation emerge from a single denoising score matching objective, replacing the heuristic KL regularization of VAE-based latent diffusion models. We prove that the resulting sample complexity depends on the intrinsic dimension rather than the ambient dimension. Experiments on Stacked MNIST, CelebA variants, and molecular generation benchmarks show that SiLD matches or outperforms VAE-based LDMs in generation quality and consistently improves reconstruction, validating our theoretical predictions.
Wei Huang, Andi Han, Mingyuan Bai +4
RIKEN AIP & The Institute of Statistical Mathematics · University of Sydney · Agency for Science, Technology and Research & The Institute of Statistical Mathematics +4
Generative modelling involves two complementary tasks: learning a probabilistic approximation of an unknown data distribution from observations, and designing a tractable mechanism for sampling from this approximation. We present a stochastic-analysis perspective on score-based generative models, connecting ideas from classical sampling theory, score estimation, and diffusion time reversal. We first explain how score functions link Langevin sampling with score matching, and how Gaussian smoothing leads to denoising score matching and multiscale sampling methods. We then emphasize the distinction between dynamics preserving a fixed distribution and dynamics transporting a prescribed family of marginals. This motivates a path-space treatment of diffusion time reversal. Finally, we connect the ideal reverse diffusion with practical generative algorithms.
Stanislas Strasman
SU, LPSM · Laboratoire de Probabilités, Statistique et Modélisation, Sorbonne Université
Most visual generative models compress images into a latent space before applying diffusion or autoregressive modelling. Yet, existing approaches such as VAEs and foundation model aligned encoders implicitly constrain the latent space without explicitly shaping its distribution, making it unclear which types of distributions are optimal for modeling. We introduce Distribution-Matching VAE (DMVAE), which explicitly aligns the encoder's latent distribution with an arbitrary reference distribution via a distribution matching constraint. This generalizes beyond the Gaussian prior of conventional VAEs, enabling alignment with distributions derived from self-supervised features, diffusion noise, or other prior distributions. With DMVAE, we can systematically investigate which latent distributions are more conducive to modeling, and we find that SSL-derived distributions provide an excellent balance between reconstruction fidelity and modeling efficiency, reaching gFID equals 3.2 on ImageNet with only 64 training epochs. Our results suggest that choosing a suitable latent distribution structure (achieved via distribution-level alignment), rather than relying on fixed priors, is key to bridging the gap between easy-to-model latents and high-fidelity image synthesis. Code is avaliable at https://github.com/sen-ye/dmvae.
Sen Ye, Jianning Pei, Mengde Xu +4
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · University of Chinese Academy of Sciences · Tencent +1