What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation

Hao Chen

stat.MLAug 25, 2026

What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation

Authors: Hao Chen

Organizations: Department of Statistics University of California, Davis

Abstract

Generative models are commonly ranked by Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), yet FID's first-two-moment summary can miss distributional differences, and a reported scalar gap alone is not a calibrated test against sampling variation. FID's moment restriction has concrete consequences: on ImageNet, visually unrecognizable images optimized only to match the reference Inception mean and covariance obtain FID $24.7$ versus $58.6$ for held-out real images (lower is better). Moreover, FID and KID are scalar discrepancies that are unchanged when the two samples are exchanged and therefore do not encode the direction of a dispersion change: under-dispersion, as can occur in mode collapse, versus over-dispersion. We introduce ZID (Z-resolved Integrated Diagnostic), which combines six standardized location- and dispersion-sensitive arms from a rank graph (RISE) and Gaussian kernels (GPK at two bandwidths). Rather than asking one scalar to serve incompatible roles, ZID reports three linked outputs: an index for ranking departure magnitude, a permutation $p$ -value for testing distributional equality, and a signed dispersion readout for diagnosis. In controlled experiments, ZID detects a broad range of departures, and its score tracks increasing severity along the corresponding sweeps, including cases in which FID is flat or reversed. On DiT-XL/2 and SiT-XL/2 guidance sweeps, ZID detects departure from real data, and its signed readout labels the high-guidance diversity collapse as under-dispersion.

Figures & tables

Panel of Figure 1 — Figure 1: Pixel-space matched-moment optimization reverses the FID ranking. The montages show 256 images from each 512-image set; metrics use all 512. Left: held-out ImageNet images. Right: noise-initialized images optimized only to match the reference Inception mean and covariance. Despite being visual noise, the optimized set receives a lower (nominally better) FID ( $24.7$ vs. $58.6$ ). ZID gives it a larger departure score ( $17.2$ vs. $0.86$ ), and none of 499 random label permutations produces an equally large score; the held-out comparison has permutation $p{=}.328$ . See Sec. 6 .

Figure 2: Controlled differences and diagnostic readouts. Two-dimensional schematics (real in blue, generated in red); FID and KID are computed from the displayed samples. All three procedures respond to a location shift. Under mean-matched scale collapse, the three scalar scores do not encode direction; ZID’s separate signed dispersion readout identifies the change as under-dispersion. For the symmetric matched-moment construction, moments through total degree three also agree, so population FID and degree- $3$ KID are both zero. ZID is not constrained to zero by either moment identity. High-dimensional finite-sample power for dispersion changes is quantified in App. A.6 .

Figure 3: Detection power across eight controlled distributional departures. Cells report empirical power at the $0.05$ significance level using real CIFAR-Inception features reduced to $d=128$ , with $n=200$ observations per sample and 300 independent repetitions. Permutation-calibrated tests use 499 relabelings. The boxed row is the complete six-arm ZID test. The next four rows report standalone RISE, GPK-med, GPK-small, and GET tests; the remaining twelve are external comparators. The eight probes span location, dispersion, dependence, higher-order structure, multimodality, and off-manifold support.

Figure 4: Signed vs. unsigned dispersion (real raw Inception-

2048

, disjoint

m{=}n{=}2500{>}d

, full-rank covariance;

Y{=}\mu{+}c(\mathrm{base}{-}\mu)

). The blue curve shows the ungated net signed dispersion score,

\operatorname{sign}(\sum_{j}Z_{D}^{(j)})S_{D}

. The formal readout

D_{\mathrm{ZID}}

equals this score when the diagnosis resolves to under- or over-dispersion; at

c=1

, the diagnosis is not assigned and

D_{\mathrm{ZID}}

is undefined (Sec. 3.4 ). FID is unsigned and is nearly equal at

c{=}0.8

and

1.1

. FID’s displayed minimum,

19.43

c{=}0.9

versus

19.50

c{=}1

, slightly favors mild collapse in this finite sample. Points are means over eight independent repetitions.

Figure 5: Ranking across sweeps of increasing severity. Spearman $\rho$ between each score and the six ordered levels of each controlled departure family. Positive values indicate that the score tends to increase as the departure becomes more severe; negative values indicate that it tends to assign smaller scores to more severe samples. Within each departure, all methods use the same embedding, sample size, and preprocessing.

	FID	KID	ZID
location	98	100	99
dispersion (collapse)	27	64	100
linear dependence	100	100	100
nonlinear dependence	45	61	92
skewness	50	61	100
kurtosis	52	48	100

Table 1: Pairwise ordering of graded departures. Accuracy is the percentage of 100 independent mild-versus-final comparisons ordered correctly; chance accuracy is

50\%

. The real CIFAR-Inception protocol is

d{=}128,n{=}200

; dispersion is diversity collapse. Bold denotes the row best.

specification	location	dispersion	lin.-dep.	nonlin.-dep.	skew	kurtosis	multimodal	off-manifold	mean
full six-arm ZID	.830	.750	.727	.980	.847	.847	.717	.833	.816
drop RISE	.843	.630	.167	.893	.347	.210	.317	.527	.492
drop GPK-med	.237	.760	.800	.983	.893	.907	.767	.870	.777
drop GPK-small	.840	.800	.740	.860	.840	.897	.697	.853	.816
$Z_{W}$ arms only	.850	.073	.793	.983	.830	.747	.777	.100	.644
$Z_{D}$ arms only	.063	.837	.127	.467	.407	.787	.130	.910	.466

Table 2: Paired construction ablation at the Fig. 3 signal levels. Results use real CIFAR-Inception features at

d=128,n=200,\alpha=.05

, with 300 repetitions and 499 outer permutations. Rejection uses

p\leq.05

. Within each repetition, every specification uses the same samples, six computed arms, and label permutations. Among the rows omitting RISE, GPK-med, or GPK-small, bold marks the largest power loss from the full six-arm ZID for each departure.

departure	setting 1	setting 2	setting 3	setting 4
common multiplier	$0.50$	$0.75$	$1.00$	$1.25$
location	$-.013$	$-.027$	$-.013$	$+.003$
dispersion	$-.040$	$\mathbf{-.060}$	$-.040$	$-.017$
linear dependence	$+.010$	$-.023$	$-.017$	$.000$
nonlinear dependence	$\mathbf{+.343}$	$\mathbf{+.193}$	$\mathbf{+.077}$	$+.013$
skewness	$.000$	$-.027$	$.000$	$+.010$

Table 3: Paired contribution of GPK-small across severity levels. Entries are paired rejection-rate differences,

\Delta=\operatorname{Power}(\text{full six-arm ZID})-\operatorname{Power}(\text{ZID without GPK-small})

. Positive values favor including GPK-small. The first seven departures use perturbation multipliers

0.50,0.75,1.00,1.25

; matched-moment multimodality replaces fractions

.25,.50,.75,1.00

of the initially real observations with their moment-matched bimodal counterparts. All cells use 300 paired repetitions, 499 outer permutations, and rejection at

p\leq.05

. Bold marks

|\Delta|\geq.05

$\varepsilon$	ZID	FID	KID	RISE	GPK-med	GPK-small
0.05	$+.97$	$-.10$	$-.13$	$+.74$	$-.10$	$\mathbf{+.98}$
0.10	$+.96$	$-.04$	$+.02$	$+.71$	$+.00$	$\mathbf{+.97}$
0.20	$+.96$	$+.04$	$-.01$	$+.66$	$-.07$	$\mathbf{+.97}$
0.30	$+.93$	$+.02$	$+.06$	$+.55$	$+.05$	$\mathbf{+.95}$
0.50	$\mathbf{+.86}$	$+.06$	$+.04$	$+.23$	$-.07$	$+.84$

Table 4: Moment-restored resampling with perturbation. Spearman

\rho

between each score and the replacement fraction

f

(real CIFAR-Inception,

d{=}128

n{=}200

), evaluated across perturbation scales

\varepsilon

(fractions of the per-coordinate standard deviations). Five perturbation scales

\varepsilon\in\{.05,.10,.20,.30,.50\}

and six replacement fractions

f\in\{0,.1,.2,.3,.4,.5\}

are evaluated with 100 repetitions at each

(\varepsilon,f)

setting. Within each repetition, every method is evaluated on the same reference sample

R

and transformed sample

B

. Their standalone columns use

T=Z_{W}^{2}+Z_{D}^{2}

. Bold marks the largest correlation in each row.

dataset	alternative	ZID	FID	KID	MIND	Density	Recall	Coverage	C2ST
CIFAR-10	collapse ( $s=.05$ )	.97	.08	.06	.18	.96	.68	.18	.03
	over-disp. ( $\tau=.25$ )	.68	.07	.07	.05	.86	.26	.39	.05
	location ( $\delta=.40$ )	.91	.67	.97	.93	.15	.05	.10	.11
	mode-drop ( $80\%$ retained; 2 classes dropped)	.85	.93	.88	.83	.06	.10	.86	.15
Tiny-IN	collapse ( $s=.05$ )	.89	.08	.05	.18	.92	.62	.15	.05
	over-disp. ( $\tau=.25$ )	.62	.07	.05	.08	.80	.27	.37	.05

Table 5: Cross-dataset comparison across four perturbations. Entries report empirical power at the

0.05

significance level on CIFAR-10 and Tiny-ImageNet (200 classes), with dataset-specific PCA-128 maps fitted to

5{,}000

Inception features,

m=n=200

, 300 repetitions, and 499 relabelings. For mode-drop, CIFAR-10 retains

80\%

of its classes and Tiny-ImageNet retains

50\%

; the other three perturbations use the same displayed strengths in both datasets. Bold marks row maxima.

Figure 6: Detection power across perturbation levels on CIFAR-10. Rows use Inception-2048 and DINO ViT-S/16 CLS embeddings; columns show variance collapse (variance-collapse amount $s$ , where deviations from the reference-feature mean are multiplied by $1-s$ ), over-dispersion (adding independent Gaussian noise with coordinate-wise standard deviation $\tau$ times the reference standard deviation), location shift (translating by $\delta$ reference standard deviations along the leading PC), and mode-drop (omitting the displayed number of classes). Curves report power at the $0.05$ significance level with $m=n=200$ and 300 repetitions per level. Permutation-calibrated methods use 99 relabelings.

generator	paradigm	domain	varied parameter	embedding	evaluation scale	role
BigGAN-deep-128	GAN	ImageNet	truncation	Inception-2048	10 classes, $n=450$	truncation ordering and direction
CIFAR DDPM	diffusion	CIFAR-10	DDIM steps	Inception-2048 + DINOv2-384	$n=200$	sampling-step comparison and direction
DiT-XL/2	diffusion transformer	ImageNet	CFG	Inception-2048	6 classes, $n=500$	over- to under-dispersion
SiT-XL/2	interpolant transformer	ImageNet	CFG	Inception-2048	6 classes, $n=500$	guidance transition in a second architecture
StyleGAN2-ADA	GAN	FFHQ faces	truncation	Inception-2048	$n=5000$	full-rank evaluation on FFHQ

Table 6: Evaluations of pretrained generative models. Each row varies only the listed sampling or truncation parameter; the pretrained generator and evaluation setup are otherwise fixed.

DDIM steps	FID	KID	ZID score	ZID test $p$ -value	dispersion diagnosis
5	173.6	.0789	6442	.002	under-dispersion
15	138.3	.0223	519	.002	member-sign conflict
25	130.8	.0148	255	.002	member-sign conflict
real–real	121.0	.0004	.109	.878	not assigned

Table 7: DDPM sampling-step sweep. A pretrained CIFAR-10 DDPM versus real CIFAR-10 in Inception

d{=}2048

m{=}n{=}200

). The ZID equality test and diagnostic gate each use 499 outer permutations. The real–real row compares disjoint halves and has no assigned dispersion direction.

Figure 7: A guidance sweep from over- to under-dispersion. Across six DiT-XL/2 ImageNet classes and

\mathrm{CFG}\in\{1,2,4,8\}

: (a) thick lines are geometric means across the six class-specific ratios to their CFG

=2

values, and shading spans one standard deviation on the log-ratio scale; (b) Precision can remain high while Recall collapses; and (c) ZID’s nonnegative

W

-component magnitude

S_{W}

tracks departure magnitude along this sweep, while its diagnostic readout

D_{\mathrm{ZID}}

flips from over-dispersed (

>0

, off-manifold) at low guidance to under-dispersed (

<0

, collapse).

metric	$\psi=1$ vs. $\psi=1$ baseline	$\psi=.9$	$\psi=.7$	$\psi=.5$
FID $\downarrow$	5.97	6.97	22.13	57.89
KID $\downarrow$	$\mathbf{-4.4{\times}10^{-5}}$	.00119	.01461	.04596
Gaussian-kernel MMD $\downarrow$	$\mathbf{-4.0\times 10^{-5}}$	.00130	.01664	.05510
Precision $\uparrow$	.817	.876	.931	.963
Density $\uparrow$	.964	1.169	1.604	1.944
Recall $\uparrow$	.750	.698	.506	.221

Table 8: StyleGAN2-ADA truncation on FFHQ. Columns compare independent draws at the indicated

\psi

with an untruncated generator reference; the

\psi=1

versus

\psi=1

baseline uses two independent untruncated draws. FID and ZID use

5000

samples; KID, Gaussian-kernel MMD, Precision, Density, and Recall use a fixed

2500

-sample subset. Arrows give the conventional preferred direction. Within each metric row, bold marks the best value for that metric. The ZID test

p

-value and

D_{\mathrm{ZID}}

are diagnostic quantities rather than ranking criteria and are not bolded. Larger ZID scores indicate stronger departures, while negative

D_{\mathrm{ZID}}

indicates under-dispersion; the

\psi=1

versus

\psi=1

baseline has no assigned direction.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset / embedding	real–real	fitted Gaussian
CIFAR-10 / Inception	2.517	1.157
Tiny-ImageNet / Inception	2.884	1.291
CIFAR-10 / DINO	76.472	33.864
Tiny-ImageNet / DINO	113.849	53.552

Appendix

Table 9: FID under fitted-Gaussian feature controls. Lower values are nominally better. The real–real column compares the reference with a held-out real sample; the Gaussian column replaces the held-out sample by a draw from the fitted Gaussian. DINO denotes the DINO ViT-S/16 CLS embeddings (

d=384

) defined in Sec. 4.5 .

Dataset	Metric	real–real	matched bimodal
CIFAR-10	FID	19.5146	0.0030
	KID	$9.5{\times}10^{-5}$	$-3.20{\times}10^{-4}$
Tiny-ImageNet	FID	25.8680	0.0024
	KID	$-5.7{\times}10^{-5}$	$-3.39{\times}10^{-4}$

Appendix

Table 10: Full-dimensional matched-bimodal feature controls. Each comparison uses raw Inception-2048 features with

m=n=2500

. The matched-bimodal sample is recolored to the reference mean and covariance. KID is the unbiased degree-

3

polynomial-kernel MMD estimate and may be slightly negative.

dataset / embedding	$\alpha{=}.01$	$.05$	$.10$
CIFAR-10 / Inception	.008	.028	.094
CIFAR-10 / DINO	.010	.052	.104
Tiny-ImageNet / Inception	.010	.054	.098
Tiny-ImageNet / DINO	.014	.048	.090

Appendix

Table 11: Outer-permutation calibration on real features. This independent real-null study applies one common protocol across four dataset–embedding settings: 500 repetitions per setting,

m=n=200

, and 99 permutations for the six-arm equality test. The CIFAR–Inception row repeats the Fig. 3 null setting as an internal reference for the matrix; the main-text check uses 999 permutations for finer

p

-value resolution. DINO denotes the DINO ViT-S/16 CLS embeddings (

d=384

) defined in Sec. 4.5 . Rejection uses

p\leq\alpha

;

p=.01

is the smallest attainable value.

departure	Gaussian-reference	permutation-tail
location	.823	.727
dispersion	.763	.763
linear dependence	.707	.643
nonlinear dependence	.977	.957
skewness	.827	.790
kurtosis	.840	.807

Appendix

Table 12: Sensitivity to arm-level tail probabilities. Each cell reports power at the

0.05

significance level over 300 paired repetitions. The Gaussian-reference specification uses the two-sided normal-reference tail of each standardized arm; the permutation-tail specification estimates each arm tail by label permutation.

Figure 8: Aligned sensitivity to Gaussian shrinkage across dimension and sample size. Empirical power for median-bandwidth GPK, degree-3 KID, and FID; $s=0$ denotes the null.

$\psi$	disp. ratio	FID	KID	ZID score	$S_{W}$	$S_{D}$	$D_{\mathrm{ZID}}$	dispersion diagnosis
1.0	0.82	90.0	.0075	328	329.1	42.2	$-42.2$	under-dispersion
0.8	0.79	88.7	.0108	566	566.4	96.1	$-96.1$	under-dispersion
0.6	0.77	96.8	.0145	813	813.8	119.3	$-119.3$	under-dispersion
0.4	0.76	101.6	.0173	1150	1151.2	176.7	$-176.7$	under-dispersion
real–real	1.00	70.3	$-.0004$	.353	1.05	.12	undefined	not assigned

Appendix

Table 13: BigGAN truncation control. BigGAN-deep-128 versus real ImageNet from the same

10

classes (Inception-2048,

m{=}n{=}450

). The dispersion ratio is

\operatorname{tr}(\widehat{\Sigma}_{\rm gen})/\operatorname{tr}(\widehat{\Sigma}_{\rm real})

. For the generator rows,

(\widehat{p}_{\mathrm{ZID}},\widehat{p}_{D})=(.002,.002)

; for real–real, the pair is

(.598,.856)

class	CFG	FID	$S_{\mathrm{ZID}}$	KID	Prec	Recall	$S_{W}$	$D_{\mathrm{ZID}}$	diagnosis ( $\widehat{p}_{D}$ )
10	1	40.9	4.1e3	0.0149	0.652	0.805	4.1e3	41.1	over (.002)
	2	20.2	3.2e3	0.0053	0.890	0.639	3.2e3	-50.3	under (.002)
	4	27.7	26e3	0.0121	0.814	0.251	26e3	-100.2	under (.002)
	8	38.4	69e3	0.0216	0.698	0.065	69e3	-148.3	under (.002)
190	1	42.3	2.2e3	0.0128	0.698	0.745	2.2e3	18.4	over (.002)
	2	35.4	6.8e3	0.0140	0.868	0.500	6.8e3	-61.3	under (.002)

Appendix

Table 14: Per-class DiT-XL/2 classifier-free-guidance sweep. Six ImageNet classes with class-matched real images (Inception-2048,

m=n=500

). Parentheses report

\widehat{p}_{D}

; “over” and “under” abbreviate the dispersion diagnoses. All

\widehat{p}_{\mathrm{ZID}}=.002

with 499 permutations.

Figure 9: Classifier-free-guidance corroboration on SiT-XL/2. Classifier-free-guidance sweep vs. real ImageNet for six classes ( $n{=}500$ ), using the 14 displayed quantities. Class 10 (left) is shown with its raw values ; the other five classes are shown as normalized color cells. Color is each row scaled by its own peak $|{\cdot}|$ within a class (diverging; green ${>}0$ , red ${<}0$ ), so color shows each method’s response shape, not cross-row magnitudes (the raw values do). The boxed first two rows place ZID and its signed dispersion readout $D_{\mathrm{ZID}}$ together; shared external methods below follow their relative order in Fig. 3 . FID, KID, and the Frobenius covariance discrepancy attain their minima near $\mathrm{CFG}{=}2$ . The specialist rows corroborate a dispersion change, while $D_{\mathrm{ZID}}$ flips positive $\,\to\,$ negative (the red band) and identifies the high-guidance side as under-dispersed; the blank cell is the one case in which no direction is assigned.

rule	power		null	Spearman $\rho$		paired accuracy
	mean	min	mean	mean	min	mean	min
flat Simes	.826	.67	.041	.829	.562	.968	.83
max $\|Z_{k}\|$	.818	.66	.042	.828	.556	.966	.82
sum of squares	.784	.60	.048	.822	.543	.958	.80
Fisher	.775	.58	.055	.812	.535	.956	.79
Cauchy	.825	.68	.042	.818	.539	.963	.80

Appendix

Table 15: Paired aggregation-rule comparison. Mean and minimum summaries are across the eight controlled departures. Power and null rejection are evaluated at the

0.05

significance level using 100 repetitions and 199 outer permutations per departure; ranking uses 100 repetitions per sweep level and per mild-versus-final pair. All rules are evaluated on identical arms within each replicate.

method	mean $\rho$	min $\rho$	mean paired accuracy	min paired accuracy
ZID	.829	.562	.968	.83
RISE	.808	.494	.969	.90
GPK-med	.411	$-.082$	.752	.47
GPK-small	.556	.087	.821	.55
GET	.679	.199	.906	.63
FID	.266	$-.406$	.659	.27

Appendix

Table 16: Expanded ranking across increasing severity. Spearman correlation uses the complete six-level sweep for each departure family; paired accuracy compares 100 independent mild-versus-final pairs per family. Mean and minimum are across the same eight departure families. MIND uses common random projections within each paired replicate, and ECS uses the independently selected fixed specification. Rows are grouped as the complete ZID construction, the standalone RISE, GPK-med, GPK-small, and GET tests, and external comparators.

multiplier	min. power	mean null	max $\|\Delta\|$	min. alt. agree	min. null agree
$.100$	.70	.035	.070	.90	.94
$.125$	.70	.041	.050	.91	.97
$.150$	.71	.039	.050	.95	.98
$.175$ (reference)	.70	.039	.000	1.00	1.00
$.200$	.70	.043	.040	.96	.98
$.225$	.70	.048	.050	.93	.98

Appendix

Table 17: Seven-point GPK-small bandwidth profile. Only the GPK-small multiplier changes; the other four arms, samples, and 199 outer relabelings are shared across ratios within each of 100 repetitions per departure. Entries give the minimum power and mean null rejection rate across departures at the

0.05

significance level. The final three columns compare each ratio with

.175

: the largest absolute departure-wise power difference and the minimum paired decision agreement under the alternative and null.

Figure 10: Detection across increasing severity. Empirical rejection rates use 300 repetitions and 499 outer permutations per cell at the

0.05

significance level; bars are Wilson

95\%

intervals. Horizontal dotted and dashed lines mark

.05

and

.80

, and the vertical dotted line marks the Fig. 3 signal. For multimodality the horizontal coordinate is the fraction of initially real observations replaced by their moment-matched bimodal counterparts; for the other seven departures it is a multiple of the Fig. 3 signal.

Figure 11: Detection across departures, sample sizes, and dimensions. The 17 rows match Fig. 3 ; columns are nine $(d,n)$ cells and panels are eight controlled departures. Each cell uses CIFAR-10 Inception features projected to $d$ pooled principal components, 100 repetitions at the $0.05$ significance level, and one signal shared by all methods; permutation-calibrated rows use 99 relabelings. The boxed row is ZID; GPK-small uses $0.175\sigma_{\rm med}$ . Right panels share the method ordering shown at left.

method	$d{=}128$			$d{=}512$			$d{=}2048$
	$n{=}50$	$n{=}200$	$n{=}600$	$n{=}50$	$n{=}200$	$n{=}600$	$n{=}50$	$n{=}200$	$n{=}600$
ZID	0.05	0.03	0.01	0.03	0.05	0.02	0.03	0.05	0.02
RISE	0.07	0.03	0.05	0.05	0.05	0.04	0.03	0.04	0.04
GPK-med	0.04	0.05	0.04	0.01	0.05	0.07	0.02	0.04	0.06
GPK-small	0.06	0.04	0.03	0.03	0.04	0.04	0.04	0.04	0.02
GET	0.05	0.05	0.05	0.03	0.05	0.04	0.08	0.06	0.06

Appendix

Table 18: Null rejection across the aligned robustness grid. The 17 rows match Fig. 11 . For each method and

(d,n)

cell, the table reports the maximum of the eight panel-specific null rejection-rate estimates, providing a worst-panel calibration summary. Each estimate uses real CIFAR-Inception features and 100 repetitions at the

0.05

significance level; rejection is defined by

p\leq.05

, and permutation-calibrated rows use 99 relabelings. The displayed values range from

.00

.10

; the binomial s.e. of each underlying estimate is approximately

.022

. GPK-small uses

0.175\sigma_{\rm med}

Generative AI Evaluation

Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.

See all 308 papers

Rethinking FID Through the Geometry of the Reference Dataset
May 28, 2026Yunghee Lee, Byeonghyun PakImage Generation Evaluation
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
Jun 18, 2026Nicolas Dufour, Alexei A. Efros, Patrick PérezML Reproducibility Image Generation Evaluation
Kyutai · UC Berkeley
MIND: Monge Inception Distance for Generative Models Evaluation
May 7, 2026Quentin Berthet, Yu-Han Wu, Clement Crepy +3Image Generation Evaluation Sliced Wasserstein Distance
Google DeepMind · LPSM, Sorbonne Université

May 28, 2026J/K move · Enter open · S save

Rethinking FID Through the Geometry of the Reference Dataset

Yunghee Lee, Byeonghyun Pak

Image Generation Evaluation

Open paper AskPDF arXiv

Fréchet Inception Distance (FID) is widely used to evaluate image generators, yet lower FID does not always correspond to better sample quality. We show that this mismatch depends in part on the geometry of the reference dataset. In a controlled study across six datasets, distributional density and effective rank significantly explain how FID changes as sample quality improves. Concentrated datasets tend to yield more favorable FID trends, whereas more dispersed datasets can make FID worsen despite better samples. Attribution to precision and recall and ablations with alternative feature spaces and distances support the same conclusion. These results suggest that distributional metrics should be interpreted together with the geometry of the reference dataset for more reliable benchmarking.

Abstract

Generative AI Evaluation

Explore similar work

Rethinking FID Through the Geometry of the Reference Dataset

The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

MIND: Monge Inception Distance for Generative Models Evaluation