Making Reconstruction FID Predictive of Diffusion Generation FID

Tongda Xu; Mingwei He; Shady Abu-Hussein; Jose Miguel Hernandez-Lobato; Chunhang Zheng; Kai Zhao; Chao Zhou; Ya-Qin Zhang

cs.CVMar 5, 2026

Making Reconstruction FID Predictive of Diffusion Generation FID

Authors: Tongda Xu, Mingwei He, Shady Abu-Hussein, Jose Miguel Hernandez-Lobato, Chunhang Zheng, Kai Zhao, Chao Zhou, Ya-Qin Zhang,

Organizations: Tsinghua University · University of Cambridge · Peking University · Kuaishou Technology

Abstract

It is well known that the reconstruction FID (rFID) of a VAE is poorly correlated with the generation FID (gFID) of a latent diffusion model. We propose interpolated FID (iFID), a simple variant of rFID that exhibits a strong correlation with gFID. Specifically, for each dataset element, we retrieve its nearest neighbor in latent space, interpolate between their latent representations, decode the interpolated latent, and compute the FID between the decoded samples and the original dataset. We provide an intuitive explanation for why iFID correlates well with gFID, and why reconstruction metrics can be negatively correlated with gFID, by connecting iFID to recent results on diffusion generalization and hallucination. Theoretically, we show that iFID evaluates decoded interpolations aligned with the ridge set around which diffusion samples concentrate, thereby measuring a quantity closely related to diffusion sample quality. Empirically, iFID is the first metric shown to strongly correlate with diffusion gFID across diverse VAEs, achieving Pearson and Spearman correlations of approximately $0.85$ . The project page is available at https://tongdaxu.github.io/pages/ifid.html.

Figures & tables

Figure 1: Left two plots : The rFID values of VAEs are uncorrelated with, or even negatively correlated with, the gFID values of diffusion models. Right two plots : The iFID metric exhibits a strong positive correlation with the gFID values of diffusion models.

Figure 2: Toy example illustrating how latent-space geometry affects diffusion samples. Left two plots : An isolated 25-component Gaussian mixture arranged on a two-dimensional square grid has poor iFID because the interpolated latent $\hat{z}$ lies off the data manifold. Diffusion samples interpolating between nearby modes also fall outside the data manifold, leading to hallucination. Right two plots : A connected 25-component Gaussian mixture has good iFID because the interpolated latent $\hat{z}$ remains on the data manifold. Diffusion samples interpolating between nearby modes therefore stay within the data manifold, reducing hallucination.

Figure 3: Ridge sets of the source distributions $p(z)$ from Figure 2 and Figure 4 , computed with $\beta=0.0$ and $t=0.1$ . The ridge sets approximate the manifolds around which diffusion samples concentrate and visually resemble lines connecting nearby training samples.

Figure 4: Toy example illustrating the difference between reconstruction-oriented and diffusion-oriented latent spaces. Left two plots : The latent distribution is an isolated two-mode Gaussian mixture with poor interpolatability, since the interpolated latent $\hat{z}$ does not lie on the data manifold. In this case, diffusion samples interpolating between nearby modes also fall outside the data manifold, leading to hallucinations. Right two plots : The latent distribution is an overlapping two-mode Gaussian mixture with good interpolatability, as the interpolated latent $\hat{z}$ remains on the data manifold. Consequently, diffusion samples interpolating between nearby modes stay within the data manifold, reducing hallucinations.

VAE Name	Latent Dim.	Arch.	Training
SD-VAE ( Rombach et al., 2022 )	$4\times 32\times 32$	UNet	Recon.
IN-VAE ( Yao and Wang, 2025 )	$32\times 16\times 16$	UNet	Recon.
FLUX-VAE ( Labs, 2024 )	$16\times 32\times 32$	UNet	Recon.
QwenImage-VAE ( Wu et al., 2025a )	$16\times 32\times 32$	UNet	Recon.
SD3-VAE ( Esser et al., 2024 )	$16\times 32\times 32$	UNet	Recon.
EQ-VAE ( Kouzelis et al., 2025 )	$4\times 32\times 32$	UNet	Recon. + Equivariance

Table 1: VAEs included in our study. We evaluate 13 VAEs with publicly available checkpoints and train corresponding diffusion models in their respective latent spaces.

Metric	gFID SiT-B				gFID SiT-XL
	w/o CFG		w/ CFG		w/o CFG		w/ CFG
	PCC $\uparrow$	SRCC $\uparrow$	PCC $\uparrow$	SRCC $\uparrow$	PCC $\uparrow$	SRCC $\uparrow$	PCC $\uparrow$	SRCC $\uparrow$
Reconstruction metrics
-PSNR	-0.81 ∗	-0.81 ∗	-0.83 ∗	-0.82 ∗	-0.79 ∗	-0.78 ∗	-0.79 ∗	-0.85 ∗
-SSIM	-0.78 ∗	-0.81 ∗	-0.80 ∗	-0.84 ∗	-0.77 ∗	-0.78 ∗	-0.77 ∗	-0.85 ∗
LPIPS	-0.73 ∗	-0.74 ∗	-0.72 ∗	-0.76 ∗	-0.73 ∗	-0.72 ∗	-0.73 ∗	-0.79 ∗

Table 2: Correlation between VAE metrics and diffusion gFID. Correlations are computed using metrics obtained from SiT models trained with 13 pre-trained VAEs. EQ Loss and SE Loss are not evaluated for VAEs with 1D latents. Following ( Zheng et al., 2025 ) , RAE is not evaluated with SiT-B. PCC denotes the Pearson correlation coefficient, and SRCC denotes Spearman’s rank correlation coefficient. Bold : best. ∗ :

p<0.05

Figure 5: Visualization of nearest-neighbor latents $\mathrm{NN}(z)$ and interpolated latents $\hat{z}$ . For reconstruction-oriented VAEs, $\mathrm{NN}(z)$ is often semantically different from $z$ , causing the decoded interpolated latents to produce invalid images. In contrast, for diffusion-oriented VAEs, $\mathrm{NN}(z)$ is semantically similar to $z$ , and the decoded interpolated latents produce realistic images.

Setting	Interpolation	# Images	$K$ for $\mathrm{NN}(\cdot)$	gFID SiT-XL
				w/o CFG		w/ CFG
				PCC $\uparrow$	SRCC $\uparrow$	PCC $\uparrow$	SRCC $\uparrow$
(a)	Linear	50k	1	0.78	0.78	0.80	0.82
(b)	Mask	50k	1	0.74	0.77	0.75	0.76
(c)	Spherical	50k	1	0.84	0.85	0.86	0.81
(d)	Spherical	200k	1	0.88	0.89	0.86	0.87

Table 3: Sensitivity analysis of the latent interpolation method, the number of images used for nearest-neighbor search, and the number of nearest neighbors

K

. iFID is robust to these choices and consistently shows a strong correlation with gFID.

Interpolation strength $\alpha$	rFID		gFID SiT-XL
	rFID		w/o CFG		w/ CFG
	PCC $\uparrow$	SRCC $\uparrow$	PCC $\uparrow$	SRCC $\uparrow$	PCC $\uparrow$	SRCC $\uparrow$
$\alpha=0.0$ (rFID)	1.00	1.00	-0.06	-0.21	-0.15	-0.31
$\alpha=0.1$	0.21	0.67	0.37	0.22	0.38	0.36
$\alpha=0.2$	-0.14	0.14	0.63	0.53	0.62	0.52
$\alpha=0.3$	-0.14	0.11	0.66	0.59	0.65	0.58

Table 4: Sensitivity analysis of the iFID interpolation strength

\alpha

. As

\alpha

increases from

0

0.5

, iFID becomes more correlated with gFID and less correlated with rFID. The correlation with gFID becomes reasonably strong when

\alpha\geq 0.2

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The relationship between the GMM loss and the diffusion loss is strong, which supports the theoretical results presented in [ 9 ] .

Panel of Figure 7 — Figure 7: The relationship between reconstruction metrics and gFID for SiT/XL. The results indicate that reconstruction metrics are often negatively correlated with gFID.

Panel of Figure 8 — Figure 8: The relationship between non-reconstruction metrics and gFID for SiT/XL. It is shown that our iFID is the only metric that is strongly correlated with gFID.

Panel of Figure 9 — Figure 9: Visualization of the decoded nearest neighbour latent NN( $z$ ) and the interpolated latent $\hat{z}$ . For reconstruction-oriented VAEs, NN( $z$ ) is semantically different from $z$ , and the interpolated $\hat{z}$ are invalid images. In contrast, for diffusion-oriented VAEs, NN( $z$ ) is semantically similar to $z$ , and the interpolated $\hat{z}$ are realistic images.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,169 papers

A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models
Sep 21, 2026Xianghong Fang, Wenjie Shu, Tongda Xu +3Distribution Shift Generative AI Evaluation
University of Toronto · Independent · Vijil
Rethinking FID Through the Geometry of the Reference Dataset
May 28, 2026Yunghee Lee, Byeonghyun PakImage Generation Evaluation
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
Jun 18, 2026Nicolas Dufour, Alexei A. Efros, Patrick PérezML Reproducibility Image Generation Evaluation
Kyutai · UC Berkeley

Sep 21, 2026J/K move · Enter open · S save

A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models

Xianghong Fang, Wenjie Shu, Tongda Xu, Wenlong Mou+2

University of Toronto · Independent · Vijil

Distribution Shift Generative AI Evaluation

Open paper AskPDF arXiv

In latent generative models, reconstruction quality is often assumed to correlate with generative performance. However, reconstruction FID (rFID) can exhibit weak or even negative correlation with generation FID (gFID). We attribute this misalignment to a latent distribution mismatch: reconstruction evaluates the decoder on encoder-induced latents, whereas generation uses the same decoder on latents produced by the generative model. To characterize this shift, we introduce generation-aware reconstruction (GAR), which constructs a continuous trajectory from standard reconstruction toward generation by perturbing encoder latents with noise and denoising them through the generative model before decoding. GAR probes the decoder behavior along this trajectory, making the transition from encoder to generation-time latent distributions observable and diagnosable. The resulting trajectory-based diagnostic, GAR-FID, exhibits strong empirical correlation with gFID across diverse tokenizers and scales. Importantly, intermediate GAR latents become more generation-aware while preserving correspondence with their source images, thereby retaining paired supervision that is absent for fully generated latents. This correspondence enables decoder adaptation on intermediate GAR latents, consistently improving generative quality across model scales. Overall, latent distribution mismatch provides a useful perspective for evaluating and improving latent generative models.

Abstract

Diffusion Models

Explore similar work

A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models

Rethinking FID Through the Geometry of the Reference Dataset

The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation