The local intrinsic dimension (LID) of data is a fundamental quantity in signal processing and learning theory, but quantifying the LID of high-dimensional, complex data has been a historically challenging task. Recent works have discovered that diffusion models capture the LID of data through the spectra of their score estimates and through the rate of change of their density estimates under various noise perturbations. While these methods can accurately quantify LID, they require either many forward passes of the diffusion model or use of gradient computation, limiting their applicability in compute- and memory-constrained scenarios. We show that the LID is a lower bound on the denoising score matching loss, motivating use of the denoising score matching loss as a LID estimator. Moreover, we show that the equivalent implicit score matching loss also approximates LID via the normal dimension and is closely related to a recent LID estimator, FLIPD. Our experiments on a manifold benchmark and with Stable Diffusion 3.5 indicate that the denoising score matching loss is a highly competitive and scalable LID estimator, achieving superior accuracy and memory footprint under increasing problem size and quantization level.
Figures & tables
Figure 1: Conceptual depiction of the denoising loss as a LID estimator (left) and a conceptual link between the denoising loss and the error bundle (EB) method.
Table 1: LID Estimate Mean Absolute Error (MAE) on Benchmark Manifolds
Figure 2: LID estimator accuracy and peak memory usage as ambient and intrinsic dimension increase on the Hypersphere manifold (5 trials). ESS uses k=50 , and DSM Loss and FLIPD use σ=0.1 .
Figure 3: FLIPD and LDSM LID estimates for 100 256×256 images using SD3.5-medium. The LID estimates are highly correlated, with LDSM providing lower estimates on average.
Figure 4: Comparison of LID prediction disagreement (MAE) and peak GPU memory usage (GB) of SD-3.5-medium at various quantization levels.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: FLIPD and LDSM LID estimates for 512×512 images using Stable Diffusion 2 (SD2). The LID estimates are highly correlated, with LDSM providing lower estimates at low noise levels.
While diffusion models enable new approaches for estimating Local Intrinsic Dimension (LID), existing methods fail in high-dimensional spaces where noise from vast normal directions overwhelms the tangent signal. We propose Local Hessian Spectral Dimension (LHSD), which resolves this by applying spectral filtering to the log-density Hessian, explicitly cutting off large eigenvalues associated with normal directions to count zero-curvature tangent directions. Implemented using Stochastic Lanczos Quadrature (SLQ), LHSD avoids full Hessian construction, achieving linear scalability with dimension D. Experiments on synthetic and real data confirm LHSD's superior robustness and its utility in detecting memorization in large-scale diffusion models. The code is available at github.com/geosada/LHSD
Diffusion models generate high-dimensional data with remarkable quality, yet how their training efficiently learns the score function, bypassing the curse of dimensionality when data is supported on low-dimensional manifolds, remains theoretically unexplained. We identify a collapse-and-refine mechanism driven by the geometry of the score function itself: at small noise scales, the diverging singularity of the score drives a rapid dimensional collapse of the induced denoising map onto the data manifold projection; at moderate noise scales, training refines the intrinsic density on the learned manifold. We instantiate this principle as Score-induced Latent Diffusion (SiLD), a two-stage framework in which both manifold learning and density estimation emerge from a single denoising score matching objective, replacing the heuristic KL regularization of VAE-based latent diffusion models. We prove that the resulting sample complexity depends on the intrinsic dimension rather than the ambient dimension. Experiments on Stacked MNIST, CelebA variants, and molecular generation benchmarks show that SiLD matches or outperforms VAE-based LDMs in generation quality and consistently improves reconstruction, validating our theoretical predictions.
Wei Huang, Andi Han, Mingyuan Bai +4
RIKEN AIP & The Institute of Statistical Mathematics · University of Sydney · Agency for Science, Technology and Research & The Institute of Statistical Mathematics +4
We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.
Anand Jerry George, Nicolas Macris
École Polytechnique Fédérale de Lausanne (EPFL), Lab for Statistical Mechanics of Inference in Large Systems (SMILS), CH-1015 Lausanne, Switzerland