Understanding Latent Diffusability via Fisher Geometry

Jing Gu; Morteza Mardani; Wonjun Lee; Dongmian Zou; Gilad Lerman

cs.LGApr 3, 2026

Understanding Latent Diffusability via Fisher Geometry

Authors: Jing Gu, Morteza Mardani, Wonjun Lee, Dongmian Zou, Gilad Lerman

Organizations: School of Mathematics, University of Minnesota · NVIDIA · Department of Mathematics, The Ohio State University · Zu Chongzhi Center and DIRC, Duke Kunshan University

Abstract

Diffusion models often degrade in latent spaces, yet the formal causes remain poorly understood. We quantify latent-space diffusability via the rate of change of the Minimum Mean Squared Error (MMSE) along the diffusion trajectory. Our framework decomposes this MMSE rate into contributions from Fisher Information (FI) and Fisher Information Rate (FIR). We show that isometric embeddings preserve intrinsic FI and establish quantitative intrinsic-FI bounds for a broader class of bi-Lipschitz encoders with controlled weak volume distortion, whereas FIR is governed by the interplay between encoder and data geometries. Our analysis separates four geometric contributions in local stability bounds for Gaussian-smoothed FIR: dimensional compression, tangential distortion, high-frequency encoder curvature, and curvature of data manifold. Experiments across diverse autoencoding architectures provide qualitative support for the geometric mechanisms identified by the theory and show that empirical FI and FIR track several measures of generation quality and latent-space geometry in the settings tested. We establish FI and FIR as a comprehensive analytical framework for understanding latent diffusability.

Figures & tables

Panel of Figure 1 — Figure 1: Values of $\mathcal{I}$ (left) and $\mathcal{R}$ (right) plotted against the noise variance $\tau$ . (a,b) Gaussian toy experiments using tiny diffusion models. Pixel denotes models trained on $\mathbf{x}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{2})$ , while latent denotes models trained on $\mathbf{z}=E(\mathbf{x})$ , with the pointwise activation $E$ indicated in the legend. (c,d) Results for recent image generative models. JiT operates directly in pixel space, whereas SiT, RAE, RAEv2, and REPA-E operate in latent space. For (c,d), we show $\sqrt{\tau}\in[0.01,80]$ , excluding smaller noise levels because of numerical instability.

Panel of Figure 2 — Figure 2: FIR deviation $\mathcal{D}_{\mathcal{R}}$ vs. (a) $\delta_{0}$ , (b) $d$ , and (c) $\varepsilon_{0}$ in toy settings. Data $\mathbf{y}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{2})$ are embedded as $\mathbf{x}=(y_{1},y_{2},0,\ldots,0)\in\mathbb{R}^{D}$ and encoded as $\mathbf{z}=E(\mathbf{x})\in\mathbb{R}^{d}$ . We compute $\mathcal{D}_{\mathcal{R}}$ from $\mathcal{R}^{(D)}(\mu_{\tau})$ and $\mathcal{R}^{(d)}((\mu_{Z})_{\tau})$ using diffusion models trained on $\mathbf{x}$ and $\mathbf{z}$ . Curves correspond to fixed noise variance $\tau$ . The solid lines $y=1.25\delta_{0}$ in (a) and $y=(D-d)/(D-2)$ in (b) serve as linear references. In (a), $D=d=2$ and $E(\mathbf{x})=\mathbf{A}\mathbf{x}$ , where $\mathbf{A}=\mathrm{diag}(\sqrt{1+\delta_{0}},\sqrt{1-\delta_{0}})$ . In (b), $D=512$ and $\mathbf{z}=(y_{1},y_{2},0,\ldots,0)\in\mathbb{R}^{d}$ . In (c), $D=d=3$ and $E((x_{1},x_{2},0),\varepsilon_{0})=\bigl(\sin(\varepsilon_{0}x_{1})/\varepsilon_{0},\,x_{2},\,(1-\cos(\varepsilon_{0}x_{1}))/\varepsilon_{0}\bigr)$ .

	Generation Metrics					Geometric and Spectral Metrics
Model	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Straightness $\uparrow$	Efficiency $\uparrow$	SEC ${}_{0.25}\downarrow$	SEC ${}_{0.5}\downarrow$
RAE	2.432	8.340	243.794	0.720	0.640	0.999	0.982	0.277	0.112
RAEv2	2.370	7.458	225.068	0.730	0.633	0.999	0.949	0.309	0.137
SiT	7.945	9.442	123.962	0.594	0.678	1.000	0.851	0.411	0.209
REPA-E	3.496	7.973	159.359	0.692	0.630	1.000	0.875	0.312	0.102
JiT	3.169	8.449	65.185	0.692	0.631	–	–	–	–

Table 1: Generation, geometric, and spectral evaluation results. Generation quality is evaluated using FID, spatial FID (sFID), Inception Score (IS), precision, and recall. Straightness and efficiency measure the geometry of ODE sampling trajectories, while SEC measures the fraction of latent spectral energy outside the low-frequency region, with subscripts indicating the frequency threshold. Geometric and spectral metrics are reported only for the latent diffusion models.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Geometric interpretation of encoder-induced distortions.

Panel of Figure 4 — Figure 4: Values of (c) $\mathcal{I}$ and (d) $\mathcal{R}$ plotted versus the noise variance $\tau$ for the ReLU-based experiments. Pixel curves correspond to models trained on $\mathbf{x}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{2})$ , while latent curves correspond to models trained on $\mathbf{z}=E(\mathbf{x})$ . For Leaky ReLU, $\alpha$ denotes the negative slope.

Panel of Figure 5 — Figure 5: FIR deviation $\mathcal{D}_{\mathcal{R}}$ vs. (a) $\tau$ and (b) $\delta_{0}$ in toy settings. Data $\mathbf{y}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{2})$ are embedded as $\mathbf{x}=(y_{1},y_{2},0,\ldots,0)\in\mathbb{R}^{D}$ and encoded to $\mathbf{z}=E(\mathbf{x})\in\mathbb{R}^{d}$ . We consider $D=d=2$ and $E(\mathbf{x})=\mathbf{A}\mathbf{x}$ with $\mathbf{A}=\mathrm{diag}(\sqrt{1+\delta_{0}},\sqrt{1-\delta_{0}})$ . We compute $\mathcal{D}_{\mathcal{R}}$ from $\mathcal{R}^{(D)}(\mu_{\tau})$ and $\mathcal{R}^{(d)}((\mu_{Z})_{\tau})$ using diffusion models trained on $\mathbf{x}$ and $\mathbf{z}$ . Solid line $y=1.25\delta_{0}$ in (b) serves as a linear reference.

Panel of Figure 6 — Figure 6: FIR deviation $\mathcal{D}_{\mathcal{R}}$ vs. (a) $\tau$ and (b) $d$ in toy settings. Data $\mathbf{y}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{2})$ are embedded as $\mathbf{x}=(y_{1},y_{2},0,\ldots,0)\in\mathbb{R}^{D}$ with $D=512$ , and $\mathbf{z}=(y_{1},y_{2},0,\ldots,0)\in\mathbb{R}^{d}$ . We compute $\mathcal{D}_{\mathcal{R}}$ from $\mathcal{R}^{(D)}(\mu_{\tau})$ and $\mathcal{R}^{(d)}((\mu_{Z})_{\tau})$ using diffusion models trained on $\mathbf{x}$ and $\mathbf{z}$ . Solid line $y=\frac{D-d}{D-2}$ in (b) serves as a linear reference.

Panel of Figure 7 — Figure 7: FIR deviation $\mathcal{D}_{\mathcal{R}}$ vs. (a) $\tau$ and (b) $\varepsilon_{0}$ in toy settings. Data $\mathbf{y}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{2})$ are embedded as $\mathbf{x}=(y_{1},y_{2},0,\ldots,0)\in\mathbb{R}^{D}$ and encoded to $\mathbf{z}=E(\mathbf{x})\in\mathbb{R}^{d}$ . We consider $D=d=3$ and $E((x_{1},x_{2},0),\varepsilon_{0})=\bigl(\sin(\varepsilon_{0}x_{1})/\varepsilon_{0},\;x_{2},\;(1-\cos(\varepsilon_{0}x_{1}))/\varepsilon_{0}\bigr)$ . We compute $\mathcal{D}_{\mathcal{R}}$ from $\mathcal{R}^{(D)}(\mu_{\tau})$ and $\mathcal{R}^{(d)}((\mu_{Z})_{\tau})$ using diffusion models trained on $\mathbf{x}$ and $\mathbf{z}$ .

Panel of Figure 8 — Figure 8: Values of (a) $\mathcal{I}$ and (b) $\mathcal{R}$ plotted versus the noise variance $\tau$ , computed from diffusion models trained on different data representations. The pixel curves correspond to models trained directly on FFHQ images. The latent curves correspond to models trained on latent representations of an image encoder (VAE or KL-AE) pretrained on FFHQ. We show $\sqrt{\tau}\in[0.01,80]$ , excluding smaller $\tau$ due to numerical instability.

Figure 9: Dimension-normalized FIR deviation $\mathcal{D}_{\mathcal{R}}^{\mathrm{sc}}$ vs. noise variance $\tau$ on FFHQ. Here, $D=64\times 64\times 3$ , $m=20$ , and the latent dimension is $d=256$ for the VAE and $d=3\times 16\times 16$ for the KL-AE. FFHQ images $\mathbf{x}$ are encoded as latents $\mathbf{z}=E(\mathbf{x})$ using the VAE or KL-AE.

Panel of Figure 10 — Figure 10: (a) Samples from a diffusion model trained directly on $64\times 64$ FFHQ. (b, c) Samples from latent diffusion models trained on (b) VAE and (c) KL-AE representations. Generated latents are mapped back to image space using their respective decoders.

Model	Reconstruction MSE
VAE	0.0094
KL-AE	0.41

Appendix

Table 2: Reconstruction MSE (mean squared error) for the VAE and KL-AE trained on FFHQ.

Model	FID vs. FFHQ	FID vs. Reconstructions
Pixel Diffusion	2.56	–
VAE (Latent Diffusion)	62.51	–
KL-AE (Latent Diffusion)	14.97	3.58

Appendix

Table 3: Generation quality on FFHQ. FID (Fréchet Inception Distance) is computed using two reference distributions: the original FFHQ images and the reconstructions produced by the corresponding autoencoder. The first FID column measures generation quality relative to the original image distribution, while the second measures generation quality relative to the reconstruction distribution of the corresponding latent model. For the pixel model, images are generated by sampling a diffusion model trained directly on FFHQ. For the VAE and KL-AE models, images are generated by sampling diffusion models trained in the latent space and decoding the generated latents using the corresponding decoder. All diffusion models use the U-Net architecture within the EDM framework.

Model	$\hat{c}$	$\hat{C}$	$\hat{C}/\hat{c}$
VAE	0.20	3.86	19.67
SD-VAE	1.60	4.83	3.02

Appendix

Table 4: Empirical estimates

(\hat{c},\hat{C})

of the constants

(c,C)

in Proposition B.1 for encoders of the VAE and SD-VAE trained on FFHQ. The ratio

\hat{C}/\hat{c}

provides an empirical estimate of the bi-Lipschitz constant.

Figure 11: Sensitivity of MMSE to $\tau$ , plotted against the noise variance $\tau$ , computed from diffusion models trained on different data representations. The pixel curve corresponds to a model trained directly on FFHQ images, while the latent curves correspond to models trained on VAE and KL-AE latent representations. We show $\sqrt{\tau}\in[0.01,80]$ , excluding smaller noise levels due to numerical instability. For visualization on a logarithmic scale, we plot $\frac{d}{d\tau}\mathrm{MMSE}+\varepsilon$ with $\varepsilon=100$ .

Panel of Figure 12 — Figure 12: Values of (a) $\mathcal{I}$ and (b) $\mathcal{R}$ plotted versus the noise variance $\tau$ for diffusion models trained on different data representations. Pixel and latent curves denote models trained on FFHQ images and their pretrained NVAE latents, respectively. NVAE was pretrained on FFHQ, with spatial size $20\times d_{\mathbf{z}}\times d_{\mathbf{z}}$ . We show $\sqrt{\tau}\in[0.01,80]$ , excluding smaller $\tau$ due to numerical instability.

Panel of Figure 13 — Figure 13: (a) Power spectrum of FFHQ images. (b)(c) Power spectra of NVAE latent representations with spatial resolutions $d_{z}=16$ and $8$ . (d)(e) Power spectra of VAE and KL-AE latent representations. The NVAE, VAE, and KL-AE models are pretrained on FFHQ images. For FFHQ images, NVAE latents, and KL-AE latents, the spectrum is computed using the two-dimensional FFT over the spatial dimensions and averaged over channels. For the one-dimensional latent vectors produced by the VAE encoder, the spectrum is computed using a one-dimensional FFT along the latent dimension. The spectra are averaged over $10{,}000$ samples drawn from the dataset. Both the frequency axis $k$ and the power values $P$ are normalized by the signal length (or spatial resolution for the 2D case). The zero-frequency (DC) component corresponding to the mean signal level is excluded.

Explore similar work

Diffusing in the Right Space: A Systematic Study of Latent Diffusability
Jun 2, 2026Tianxiong Zhong, Xingye Tian, Xuebo Wang +2Visual Tokenizers Diffusion Dynamics
Kling Team, Kuaishou Technology
Geometric Decoupling: Diagnosing the Structural Instability of Latent
Apr 20, 2026Yuanbang Liang, Zhengwen Chen, Yu-Kun LaiDiscriminative Congruence Transform Instability
School of Computer Science and Informatics, Cardiff University, Cardiff, United Kingdom
What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion
May 8, 2026Zhengrong Yue, Taihang Hu, Mengting Chen +8Latent Diffusion Model Diffusion Models
Shanghai Jiao Tong University · Alibaba Group · Beihang University +4

Jun 2, 2026, cs.CVJ/K move · Enter open · S save

Diffusing in the Right Space: A Systematic Study of Latent Diffusability

Tianxiong Zhong, Xingye Tian, Xuebo Wang, Xin Tao+1

Kling Team, Kuaishou Technology

Visual Tokenizers Diffusion Dynamics Generative Models Systematic Study

Open paper AskPDF arXiv

Latent diffusion models leverage visual tokenizers to compress images into latent spaces for efficient generative modeling. However, better reconstruction quality of a tokenizer does not necessarily translate into better generation quality, suggesting that latent representations should be evaluated not only by fidelity but also by their diffusability. Recent studies have proposed diverse explanations for diffusion-friendly latent spaces, including semantic separability, affine equivariance, distribution uniformity, spatial structure, spectral smoothness, and manifold continuity. Yet these properties are often validated on a limited set of tokenizers, leaving it unclear which factors are most predictive of downstream generation quality and whether such conclusions hold beyond the specific settings in which they are introduced. In this work, we conduct a systematic study of latent diffusability by training a large collection of tokenizers with diverse regularization strategies, architectures, and latent configurations, and evaluating them with multiple downstream diffusion backbones. Our analysis identifies several latent properties that consistently correlate with generation quality and exhibit strong generalization across experimental settings. Beyond existing metrics, we introduce Velocity Irreducible Variance (VIV), a measure of velocity ambiguity induced by trajectory crossings. Extensive experiments show that VIV is one of the most stable predictors of generation quality.

Abstract

Explore similar work

Diffusing in the Right Space: A Systematic Study of Latent Diffusability

Geometric Decoupling: Diagnosing the Structural Instability of Latent

What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion