Generative models are commonly ranked by Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), yet FID's first-two-moment summary can miss distributional differences, and a reported scalar gap alone is not a calibrated test against sampling variation. FID's moment restriction has concrete consequences: on ImageNet, visually unrecognizable images optimized only to match the reference Inception mean and covariance obtain FID 24.7 versus 58.6 for held-out real images (lower is better). Moreover, FID and KID are scalar discrepancies that are unchanged when the two samples are exchanged and therefore do not encode the direction of a dispersion change: under-dispersion, as can occur in mode collapse, versus over-dispersion. We introduce ZID (Z-resolved Integrated Diagnostic), which combines six standardized location- and dispersion-sensitive arms from a rank graph (RISE) and Gaussian kernels (GPK at two bandwidths). Rather than asking one scalar to serve incompatible roles, ZID reports three linked outputs: an index for ranking departure magnitude, a permutation p-value for testing distributional equality, and a signed dispersion readout for diagnosis. In controlled experiments, ZID detects a broad range of departures, and its score tracks increasing severity along the corresponding sweeps, including cases in which FID is flat or reversed. On DiT-XL/2 and SiT-XL/2 guidance sweeps, ZID detects departure from real data, and its signed readout labels the high-guidance diversity collapse as under-dispersion.
Figures & tables
Figure 1: Pixel-space matched-moment optimization reverses the FID ranking. The montages show 256 images from each 512-image set; metrics use all 512. Left: held-out ImageNet images. Right: noise-initialized images optimized only to match the reference Inception mean and covariance. Despite being visual noise, the optimized set receives a lower (nominally better) FID ( 24.7 vs. 58.6 ). ZID gives it a larger departure score ( 17.2 vs. 0.86 ), and none of 499 random label permutations produces an equally large score; the held-out comparison has permutation p=.328 . See Sec. 6 .
Figure 2: Controlled differences and diagnostic readouts. Two-dimensional schematics (real in blue, generated in red); FID and KID are computed from the displayed samples. All three procedures respond to a location shift. Under mean-matched scale collapse, the three scalar scores do not encode direction; ZID’s separate signed dispersion readout identifies the change as under-dispersion. For the symmetric matched-moment construction, moments through total degree three also agree, so population FID and degree- 3 KID are both zero. ZID is not constrained to zero by either moment identity. High-dimensional finite-sample power for dispersion changes is quantified in App. A.6 .
Figure 3: Detection power across eight controlled distributional departures. Cells report empirical power at the 0.05 significance level using real CIFAR-Inception features reduced to d=128 , with n=200 observations per sample and 300 independent repetitions. Permutation-calibrated tests use 499 relabelings. The boxed row is the complete six-arm ZID test. The next four rows report standalone RISE, GPK-med, GPK-small, and GET tests; the remaining twelve are external comparators. The eight probes span location, dispersion, dependence, higher-order structure, multimodality, and off-manifold support.
Figure 4: Signed vs. unsigned dispersion (real raw Inception- 2048 , disjoint m=n=2500>d , full-rank covariance; Y=μ+c(base−μ) ). The blue curve shows the ungated net signed dispersion score, sign(∑jZD(j))SD . The formal readout DZID equals this score when the diagnosis resolves to under- or over-dispersion; at c=1 , the diagnosis is not assigned and DZID is undefined (Sec. 3.4 ). FID is unsigned and is nearly equal at c=0.8 and 1.1 . FID’s displayed minimum, 19.43 at c=0.9 versus 19.50 at c=1 , slightly favors mild collapse in this finite sample. Points are means over eight independent repetitions.
Figure 5: Ranking across sweeps of increasing severity. Spearman ρ between each score and the six ordered levels of each controlled departure family. Positive values indicate that the score tends to increase as the departure becomes more severe; negative values indicate that it tends to assign smaller scores to more severe samples. Within each departure, all methods use the same embedding, sample size, and preprocessing.
FID
KID
ZID
location
98
100
99
dispersion (collapse)
27
64
100
linear dependence
100
100
100
nonlinear dependence
45
61
92
skewness
50
61
100
kurtosis
52
48
100
Table 1: Pairwise ordering of graded departures. Accuracy is the percentage of 100 independent mild-versus-final comparisons ordered correctly; chance accuracy is 50% . The real CIFAR-Inception protocol is d=128,n=200 ; dispersion is diversity collapse. Bold denotes the row best.
specification
location
dispersion
lin.-dep.
nonlin.-dep.
skew
kurtosis
multimodal
off-manifold
mean
full six-arm ZID
.830
.750
.727
.980
.847
.847
.717
.833
.816
drop RISE
.843
.630
.167
.893
.347
.210
.317
.527
.492
drop GPK-med
.237
.760
.800
.983
.893
.907
.767
.870
.777
drop GPK-small
.840
.800
.740
.860
.840
.897
.697
.853
.816
ZW arms only
.850
.073
.793
.983
.830
.747
.777
.100
.644
ZD arms only
.063
.837
.127
.467
.407
.787
.130
.910
.466
Table 2: Paired construction ablation at the Fig. 3 signal levels. Results use real CIFAR-Inception features at d=128,n=200,α=.05 , with 300 repetitions and 499 outer permutations. Rejection uses p≤.05 . Within each repetition, every specification uses the same samples, six computed arms, and label permutations. Among the rows omitting RISE, GPK-med, or GPK-small, bold marks the largest power loss from the full six-arm ZID for each departure.
departure
setting 1
setting 2
setting 3
setting 4
common multiplier
0.50
0.75
1.00
1.25
location
−.013
−.027
−.013
+.003
dispersion
−.040
−.060
−.040
−.017
linear dependence
+.010
−.023
−.017
.000
nonlinear dependence
+.343
+.193
+.077
+.013
skewness
.000
−.027
.000
+.010
Table 3: Paired contribution of GPK-small across severity levels. Entries are paired rejection-rate differences, Δ=Power(full six-arm ZID)−Power(ZID without GPK-small) . Positive values favor including GPK-small. The first seven departures use perturbation multipliers 0.50,0.75,1.00,1.25 ; matched-moment multimodality replaces fractions .25,.50,.75,1.00 of the initially real observations with their moment-matched bimodal counterparts. All cells use 300 paired repetitions, 499 outer permutations, and rejection at p≤.05 . Bold marks ∣Δ∣≥.05 .
ε
ZID
FID
KID
RISE
GPK-med
GPK-small
0.05
+.97
−.10
−.13
+.74
−.10
+.98
0.10
+.96
−.04
+.02
+.71
+.00
+.97
0.20
+.96
+.04
−.01
+.66
−.07
+.97
0.30
+.93
+.02
+.06
+.55
+.05
+.95
0.50
+.86
+.06
+.04
+.23
−.07
+.84
Table 4: Moment-restored resampling with perturbation. Spearman ρ between each score and the replacement fraction f (real CIFAR-Inception, d=128 , n=200 ), evaluated across perturbation scales ε (fractions of the per-coordinate standard deviations). Five perturbation scales ε∈{.05,.10,.20,.30,.50} and six replacement fractions f∈{0,.1,.2,.3,.4,.5} are evaluated with 100 repetitions at each (ε,f) setting. Within each repetition, every method is evaluated on the same reference sample R and transformed sample B . Their standalone columns use T=ZW2+ZD2 . Bold marks the largest correlation in each row.
dataset
alternative
ZID
FID
KID
MIND
Density
Recall
Coverage
C2ST
CIFAR-10
collapse ( s=.05 )
.97
.08
.06
.18
.96
.68
.18
.03
over-disp. ( τ=.25 )
.68
.07
.07
.05
.86
.26
.39
.05
location ( δ=.40 )
.91
.67
.97
.93
.15
.05
.10
.11
mode-drop ( 80% retained; 2 classes dropped)
.85
.93
.88
.83
.06
.10
.86
.15
Tiny-IN
collapse ( s=.05 )
.89
.08
.05
.18
.92
.62
.15
.05
over-disp. ( τ=.25 )
.62
.07
.05
.08
.80
.27
.37
.05
Table 5: Cross-dataset comparison across four perturbations. Entries report empirical power at the 0.05 significance level on CIFAR-10 and Tiny-ImageNet (200 classes), with dataset-specific PCA-128 maps fitted to 5,000 Inception features, m=n=200 , 300 repetitions, and 499 relabelings. For mode-drop, CIFAR-10 retains 80% of its classes and Tiny-ImageNet retains 50% ; the other three perturbations use the same displayed strengths in both datasets. Bold marks row maxima.
Figure 6: Detection power across perturbation levels on CIFAR-10. Rows use Inception-2048 and DINO ViT-S/16 CLS embeddings; columns show variance collapse (variance-collapse amount s , where deviations from the reference-feature mean are multiplied by 1−s ), over-dispersion (adding independent Gaussian noise with coordinate-wise standard deviation τ times the reference standard deviation), location shift (translating by δ reference standard deviations along the leading PC), and mode-drop (omitting the displayed number of classes). Curves report power at the 0.05 significance level with m=n=200 and 300 repetitions per level. Permutation-calibrated methods use 99 relabelings.
generator
paradigm
domain
varied parameter
embedding
evaluation scale
role
BigGAN-deep-128
GAN
ImageNet
truncation
Inception-2048
10 classes, n=450
truncation ordering and direction
CIFAR DDPM
diffusion
CIFAR-10
DDIM steps
Inception-2048 + DINOv2-384
n=200
sampling-step comparison and direction
DiT-XL/2
diffusion transformer
ImageNet
CFG
Inception-2048
6 classes, n=500
over- to under-dispersion
SiT-XL/2
interpolant transformer
ImageNet
CFG
Inception-2048
6 classes, n=500
guidance transition in a second architecture
StyleGAN2-ADA
GAN
FFHQ faces
truncation
Inception-2048
n=5000
full-rank evaluation on FFHQ
Table 6: Evaluations of pretrained generative models. Each row varies only the listed sampling or truncation parameter; the pretrained generator and evaluation setup are otherwise fixed.
DDIM steps
FID
KID
ZID score
ZID test p -value
dispersion diagnosis
5
173.6
.0789
6442
.002
under-dispersion
15
138.3
.0223
519
.002
member-sign conflict
25
130.8
.0148
255
.002
member-sign conflict
real–real
121.0
.0004
.109
.878
not assigned
Table 7: DDPM sampling-step sweep. A pretrained CIFAR-10 DDPM versus real CIFAR-10 in Inception d=2048 , m=n=200 ). The ZID equality test and diagnostic gate each use 499 outer permutations. The real–real row compares disjoint halves and has no assigned dispersion direction.
Figure 7: A guidance sweep from over- to under-dispersion. Across six DiT-XL/2 ImageNet classes and CFG∈{1,2,4,8} : (a) thick lines are geometric means across the six class-specific ratios to their CFG =2 values, and shading spans one standard deviation on the log-ratio scale; (b) Precision can remain high while Recall collapses; and (c) ZID’s nonnegative W -component magnitude SW tracks departure magnitude along this sweep, while its diagnostic readout DZID flips from over-dispersed ( >0 , off-manifold) at low guidance to under-dispersed ( <0 , collapse).
metric
ψ=1 vs. ψ=1 baseline
ψ=.9
ψ=.7
ψ=.5
FID ↓
5.97
6.97
22.13
57.89
KID ↓
−4.4×10−5
.00119
.01461
.04596
Gaussian-kernel MMD ↓
−4.0×10−5
.00130
.01664
.05510
Precision ↑
.817
.876
.931
.963
Density ↑
.964
1.169
1.604
1.944
Recall ↑
.750
.698
.506
.221
Table 8: StyleGAN2-ADA truncation on FFHQ. Columns compare independent draws at the indicated ψ with an untruncated generator reference; the ψ=1 versus ψ=1 baseline uses two independent untruncated draws. FID and ZID use 5000 samples; KID, Gaussian-kernel MMD, Precision, Density, and Recall use a fixed 2500 -sample subset. Arrows give the conventional preferred direction. Within each metric row, bold marks the best value for that metric. The ZID test p -value and DZID are diagnostic quantities rather than ranking criteria and are not bolded. Larger ZID scores indicate stronger departures, while negative DZID indicates under-dispersion; the ψ=1 versus ψ=1 baseline has no assigned direction.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset / embedding
real–real
fitted Gaussian
CIFAR-10 / Inception
2.517
1.157
Tiny-ImageNet / Inception
2.884
1.291
CIFAR-10 / DINO
76.472
33.864
Tiny-ImageNet / DINO
113.849
53.552
Appendix
Table 9: FID under fitted-Gaussian feature controls. Lower values are nominally better. The real–real column compares the reference with a held-out real sample; the Gaussian column replaces the held-out sample by a draw from the fitted Gaussian. DINO denotes the DINO ViT-S/16 CLS embeddings ( d=384 ) defined in Sec. 4.5 .
Dataset
Metric
real–real
matched bimodal
CIFAR-10
FID
19.5146
0.0030
KID
9.5×10−5
−3.20×10−4
Tiny-ImageNet
FID
25.8680
0.0024
KID
−5.7×10−5
−3.39×10−4
Appendix
Table 10: Full-dimensional matched-bimodal feature controls. Each comparison uses raw Inception-2048 features with m=n=2500 . The matched-bimodal sample is recolored to the reference mean and covariance. KID is the unbiased degree- 3 polynomial-kernel MMD estimate and may be slightly negative.
dataset / embedding
α=.01
.05
.10
CIFAR-10 / Inception
.008
.028
.094
CIFAR-10 / DINO
.010
.052
.104
Tiny-ImageNet / Inception
.010
.054
.098
Tiny-ImageNet / DINO
.014
.048
.090
Appendix
Table 11: Outer-permutation calibration on real features. This independent real-null study applies one common protocol across four dataset–embedding settings: 500 repetitions per setting, m=n=200 , and 99 permutations for the six-arm equality test. The CIFAR–Inception row repeats the Fig. 3 null setting as an internal reference for the matrix; the main-text check uses 999 permutations for finer p -value resolution. DINO denotes the DINO ViT-S/16 CLS embeddings ( d=384 ) defined in Sec. 4.5 . Rejection uses p≤α ; p=.01 is the smallest attainable value.
departure
Gaussian-reference
permutation-tail
location
.823
.727
dispersion
.763
.763
linear dependence
.707
.643
nonlinear dependence
.977
.957
skewness
.827
.790
kurtosis
.840
.807
Appendix
Table 12: Sensitivity to arm-level tail probabilities. Each cell reports power at the 0.05 significance level over 300 paired repetitions. The Gaussian-reference specification uses the two-sided normal-reference tail of each standardized arm; the permutation-tail specification estimates each arm tail by label permutation.
Figure 8: Aligned sensitivity to Gaussian shrinkage across dimension and sample size. Empirical power for median-bandwidth GPK, degree-3 KID, and FID; s=0 denotes the null.
ψ
disp. ratio
FID
KID
ZID score
SW
SD
DZID
dispersion diagnosis
1.0
0.82
90.0
.0075
328
329.1
42.2
−42.2
under-dispersion
0.8
0.79
88.7
.0108
566
566.4
96.1
−96.1
under-dispersion
0.6
0.77
96.8
.0145
813
813.8
119.3
−119.3
under-dispersion
0.4
0.76
101.6
.0173
1150
1151.2
176.7
−176.7
under-dispersion
real–real
1.00
70.3
−.0004
.353
1.05
.12
undefined
not assigned
Appendix
Table 13: BigGAN truncation control. BigGAN-deep-128 versus real ImageNet from the same 10 classes (Inception-2048, m=n=450 ). The dispersion ratio is tr(Σgen)/tr(Σreal) . For the generator rows, (pZID,pD)=(.002,.002) ; for real–real, the pair is (.598,.856) .
class
CFG
FID
SZID
KID
Prec
Recall
SW
DZID
diagnosis ( pD )
10
1
40.9
4.1e3
0.0149
0.652
0.805
4.1e3
41.1
over (.002)
2
20.2
3.2e3
0.0053
0.890
0.639
3.2e3
-50.3
under (.002)
4
27.7
26e3
0.0121
0.814
0.251
26e3
-100.2
under (.002)
8
38.4
69e3
0.0216
0.698
0.065
69e3
-148.3
under (.002)
190
1
42.3
2.2e3
0.0128
0.698
0.745
2.2e3
18.4
over (.002)
2
35.4
6.8e3
0.0140
0.868
0.500
6.8e3
-61.3
under (.002)
Appendix
Table 14: Per-class DiT-XL/2 classifier-free-guidance sweep. Six ImageNet classes with class-matched real images (Inception-2048, m=n=500 ). Parentheses report pD ; “over” and “under” abbreviate the dispersion diagnoses. All pZID=.002 with 499 permutations.
Figure 9: Classifier-free-guidance corroboration on SiT-XL/2. Classifier-free-guidance sweep vs. real ImageNet for six classes ( n=500 ), using the 14 displayed quantities. Class 10 (left) is shown with its raw values ; the other five classes are shown as normalized color cells. Color is each row scaled by its own peak ∣⋅∣ within a class (diverging; green >0 , red <0 ), so color shows each method’s response shape, not cross-row magnitudes (the raw values do). The boxed first two rows place ZID and its signed dispersion readout DZID together; shared external methods below follow their relative order in Fig. 3 . FID, KID, and the Frobenius covariance discrepancy attain their minima near CFG=2 . The specialist rows corroborate a dispersion change, while DZID flips positive → negative (the red band) and identifies the high-guidance side as under-dispersed; the blank cell is the one case in which no direction is assigned.
rule
power
null
Spearman ρ
paired accuracy
mean
min
mean
mean
min
mean
min
flat Simes
.826
.67
.041
.829
.562
.968
.83
max ∣Zk∣
.818
.66
.042
.828
.556
.966
.82
sum of squares
.784
.60
.048
.822
.543
.958
.80
Fisher
.775
.58
.055
.812
.535
.956
.79
Cauchy
.825
.68
.042
.818
.539
.963
.80
Appendix
Table 15: Paired aggregation-rule comparison. Mean and minimum summaries are across the eight controlled departures. Power and null rejection are evaluated at the 0.05 significance level using 100 repetitions and 199 outer permutations per departure; ranking uses 100 repetitions per sweep level and per mild-versus-final pair. All rules are evaluated on identical arms within each replicate.
method
mean ρ
min ρ
mean paired accuracy
min paired accuracy
ZID
.829
.562
.968
.83
RISE
.808
.494
.969
.90
GPK-med
.411
−.082
.752
.47
GPK-small
.556
.087
.821
.55
GET
.679
.199
.906
.63
FID
.266
−.406
.659
.27
Appendix
Table 16: Expanded ranking across increasing severity. Spearman correlation uses the complete six-level sweep for each departure family; paired accuracy compares 100 independent mild-versus-final pairs per family. Mean and minimum are across the same eight departure families. MIND uses common random projections within each paired replicate, and ECS uses the independently selected fixed specification. Rows are grouped as the complete ZID construction, the standalone RISE, GPK-med, GPK-small, and GET tests, and external comparators.
multiplier
min. power
mean null
max ∣Δ∣
min. alt. agree
min. null agree
.100
.70
.035
.070
.90
.94
.125
.70
.041
.050
.91
.97
.150
.71
.039
.050
.95
.98
.175 (reference)
.70
.039
.000
1.00
1.00
.200
.70
.043
.040
.96
.98
.225
.70
.048
.050
.93
.98
Appendix
Table 17: Seven-point GPK-small bandwidth profile. Only the GPK-small multiplier changes; the other four arms, samples, and 199 outer relabelings are shared across ratios within each of 100 repetitions per departure. Entries give the minimum power and mean null rejection rate across departures at the 0.05 significance level. The final three columns compare each ratio with .175 : the largest absolute departure-wise power difference and the minimum paired decision agreement under the alternative and null.
Figure 10: Detection across increasing severity. Empirical rejection rates use 300 repetitions and 499 outer permutations per cell at the 0.05 significance level; bars are Wilson 95% intervals. Horizontal dotted and dashed lines mark .05 and .80 , and the vertical dotted line marks the Fig. 3 signal. For multimodality the horizontal coordinate is the fraction of initially real observations replaced by their moment-matched bimodal counterparts; for the other seven departures it is a multiple of the Fig. 3 signal.
Figure 11: Detection across departures, sample sizes, and dimensions. The 17 rows match Fig. 3 ; columns are nine (d,n) cells and panels are eight controlled departures. Each cell uses CIFAR-10 Inception features projected to d pooled principal components, 100 repetitions at the 0.05 significance level, and one signal shared by all methods; permutation-calibrated rows use 99 relabelings. The boxed row is ZID; GPK-small uses 0.175σmed . Right panels share the method ordering shown at left.
Figure 29
method
d=128
d=512
d=2048
n=50
n=200
n=600
n=50
n=200
n=600
n=50
n=200
n=600
ZID
0.05
0.03
0.01
0.03
0.05
0.02
0.03
0.05
0.02
RISE
0.07
0.03
0.05
0.05
0.05
0.04
0.03
0.04
0.04
GPK-med
0.04
0.05
0.04
0.01
0.05
0.07
0.02
0.04
0.06
GPK-small
0.06
0.04
0.03
0.03
0.04
0.04
0.04
0.04
0.02
GET
0.05
0.05
0.05
0.03
0.05
0.04
0.08
0.06
0.06
Appendix
Table 18: Null rejection across the aligned robustness grid. The 17 rows match Fig. 11 . For each method and (d,n) cell, the table reports the maximum of the eight panel-specific null rejection-rate estimates, providing a worst-panel calibration summary. Each estimate uses real CIFAR-Inception features and 100 repetitions at the 0.05 significance level; rejection is defined by p≤.05 , and permutation-calibrated rows use 99 relabelings. The displayed values range from .00 to .10 ; the binomial s.e. of each underlying estimate is approximately .022 . GPK-small uses 0.175σmed .
Fréchet Inception Distance (FID) is widely used to evaluate image generators, yet lower FID does not always correspond to better sample quality. We show that this mismatch depends in part on the geometry of the reference dataset. In a controlled study across six datasets, distributional density and effective rank significantly explain how FID changes as sample quality improves. Concentrated datasets tend to yield more favorable FID trends, whereas more dispersed datasets can make FID worsen despite better samples. Attribution to precision and recall and ablations with alternative feature spaces and distances support the same conclusion. These results suggest that distributional metrics should be interpreted together with the geometry of the reference dataset for more reliable benchmarking.
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.
We propose the Monge Inception Distance (MIND), a metric for evaluating generative models that addresses key limitations of the widely adopted Fréchet Inception Distance (FID). The MIND metric leverages the sliced Wasserstein distance to compare distributions by averaging one-dimensional optimal transport distances, efficiently computed via sorting. This approach circumvents the estimation of high-dimensional means and covariance matrices, which underlie FID's poor sample complexity and vulnerability to adversarial attacks. We empirically demonstrate three primary advantages: (i) it is more sample-efficient by one order of magnitude, (ii) it is faster to compute by two orders of magnitude, (iii) it is more robust to adversarial attacks such as moment-matching. We show that MIND with 5k samples can replace the evaluation performance of FID with 50k samples, providing high correlation with this standard benchmark and superior discriminative performance. We further demonstrate that even smaller sample sizes (e.g., 1k or 2k) remain highly informative for rapid model iteration.